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Abstract 

The model checking problem for various fragments of first-order logic has attracted much atten- 
tion over the last two decades: in particular, for the fragment induced by 3 and A and that induced 
by V, 3 and A, which are better known as the constraint satisfaction problem and the quantified con- 
straint satisfaction problem, respectively. These two fragments are in fact the only ones for which 
there is currently no known complexity classification. All other syntactic fragments can be easily 
classified, either directly or using Schaefer's dichotomy theorems for SAT and QSAT, with the ex- 
ception of the positive equality free fragment induced by 3,V, A and V. This outstanding fragment 
can also be classified and enjoys a tetrachotomy: according to the model, the corresponding model 
checking problem is either tractable, NP-complete, co-NP-complete or Pspace-complete. Moreover, 
the complexity drop is always witnessed by a generic solving algorithm which uses quantifier relativi- 
sation (for example, in the co-NP-complete case, the model has a constant e to which all 3 quantifiers 
may be relativised). Furthermore, its complexity is characterised by algebraic means: the presence 
or absence of specific surjective hyper-operations among those that preserve the model characterise 
the complexity. Our classification methodology relies on this suitably tailored algebraic approach 
and it suffices to classify the complexity of a finite number of cases for each model size: each case 
corresponds to an element of the finite lattice of down-closed monoids of surjective hyper operations. 
This is unlike the constraint satisfaction problem where the corresponding lattices are uncountable 
and essentially uncharted in general. Though finite, the number of elements of the lattice of down- 
closed monoids of surjective hyper operations grows rapidly as the size n of the model increases. We 
are able to compute suitable parts of this lattice by hand for n = 2 and 3, and in a computer assisted 
manner for n — 4. For arbitrarily large n, one can restrict the classification to specific monoids of 
surjective hyper-operations which corresponds to certain cores, for a suitable notion of core for the 
positive equality-free fragment of first order logic. These specific monoids enjoy a nice normal form 
which means that we are able to provide generic hardness proofs which mimic cases encountered 
when n < 4. 

Keywords: Constraint Satisfaction, Galois Connection, Logic in Computer Science, Quantified Con- 
straint Satisfaction, Universal Algebra. 



1 Introduction 

The model checking problem over a logic ^ takes as input a structure ^ and a sentence <p of and 
asks whether &\=(p. The problem can also be parameterised, either by the sentence (p, in which case the 
input is simply or by the model in which case the input is simply (p. Vardi has studied the com- 



plexity of this problem, principly for logics which subsume FO | Var82 1 . He describes the complexity of 
the unrestricted problem as the combined complexity, and the complexity of the parameterisation by the 
sentence (respectively, model) as the data complexity (respectively, expression complexity). For the ma- 
jority of his logics, the expression and combined complexities are comparable, and are one exponential 
higher than the data complexity. 

In this paper, we will be interested in taking syntactic fragments of FO, induced by the presence 
or absence of quantifiers and connectives, and studying the complexities of the parameterisation of the 
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model checking problem by the model ^, that is the expression complexities for certain When 
^ is the primitive positive fragment of FO, {3, A}-FO, the model checking problem is equivalent to 
the much-studied constraint satisfaction problem (CSP). The parameterisation of this problem by the 
model S) is equivalent to what is sometimes described as the non-uniform constraint satisfaction problem, 
CSP(^) IKVOOJ. It has been conjectured | ,BKJ00,FV9 8 | that the class of CSPs exhibits dichotomy - that 
is, CSP(^) is always either in P or is NP-complete, depending on the model Qi. This is tantamount to 
the condition that the expression complexity for {A, 3}-F0 on Q is always either in P or is NP-complete. 
While in general this conjecture remains open, it has been proved for substantial classes and various 
methods, combinatorial (graph-theoretic), logical and universal-algebraic have been brought to bear on 
this classification project, with many remarkable consequences. Schaefer was a precursor and provided 
a dichotomy for Boolean structures using a logico-combinatorial approach | |Sch78[ . Further dichotomies 
were obtained: e.g. for structures of size at most three |Bul06 ], for undirected graphs fHN90|, smooth 
digraphs |jBKN09J. A conjectured delineation for the dichotomy was given in the algebraic language 
in 0BJKO5I . 

When ^ is positive Horn, {3,V, A}-FO, the model checking problem is equivalent to the well- 
studied quantified constraint satisfaction problem (QCSP). No overarching polychotomy has been con- 
jectured for the non-uniform QCSP(^), although the only known attainable complexities are P, NP- 
complete and Pspace-complete. Schaefer announced a dichotomy in the Boolean case |Sch78| be- 
tween P and Pspace-complete in the presence of constants, a dichotomy which was proved to hold 
even when constants are not present |Dal97[ CKSOl |. Some partial classification were obtained, al- 
gebraically | |Che04[[Che08„BBC+09] or combinatorially | |MM06„Marll] . A conjecture delineating the 
border between NP and Pspace-complete was recently ventured by Chen in the algebraic language for 
structures with all constants | |Chel2| . 

Owing to the natural duality between 3,V and V, A, we consider also various dual fragments. For 
example, the dual of {3, A}-FO is positive universal disjunctive FO, {V, V}-FO. It is straightforward to 
see that this class of expression complexities exhibits dichotomy between P and co-NP-complete if, and 
only if, the class of CSPs exhibits dichotomy between P and NP-complete. Table [T] summarises known 
results regarding the complexity of the model checking for syntactic fragments of first-order logic, up to 
this duality. 

In the case of primitive positive logic, it makes little difference whether or not equality is allowed, that 
is the expression complexities for {3, A}-FO and {3, A,=}-FO are equivalent. This is because equality 
may be propagated out in all but trivial instances. The same is true for positive Horn logic, but is not 
true, e.g., for positive universal disjunctive FO. Indeed, a classification of the expression complexities 
over {V,V}-FO is equivalent to the unproven CSP dichotomy conjecture, though we are able to give a 
full dichotomy for the expression complexities over {V, V, =}-F0. The reason for this is that the equality 
relation in the latter simulates a disequality relation in the former. If the model 'S has ^ > 3 elements 
then {3, A, 7^}-F0 can simulate ^-colourability; and, otherwise we have a Boolean model and Schaefer's 
dichotomy theorem provides the classification. A similar phenomenon occurs at a higher level when V is 
also present. 

Other fragments can be easily classified, as the model checking problem is always hard except for 
pathological and rather trivial models, with the notable exception of positive equality-free first-order 
logic {3, V, A, V}-FO. For this outstanding fragment, the corresponding model checking problem can 
be seen as an extension of QCSP in which disjunction is returned to the mix. Note that the absence of 
equality is here important, as there is no general method for its being propagated out by substitution. 
Indeed, we will see that evaluating the related fragment {3,V, A, V, =}-F0 is Pspace-complete on any 
structure ^ of size at least two. 

The case of {3, V, A, V}-FO is considerably richer than all other cases (as seen on Table [TJ - with 
the exception of {3,A}-F0 and {3,V,A}-F0 which are still open and active fields of research - and 



2 



On the complexity of the model checking problem 



F.R. Madelaine and B.D. Martin 



Fragment 


Dual 


Classification? 


{3,V} 
{3,V,=} 


{V,A} 
{V,A,/} 


Trivial (in L). 


{3,A,V} 
{3,A,V,=} 


{V,A,V} 
{V,A,V,/} 


Trivial (in L) if the core of S) has one element and 
NP-complete otherwise. 


{3,A,V,7^} 


{V,A,V,=} 


Trivial (in L) if D = 1 and NP-complete otherwise. 


{3,A} 
{3,A,=} 


{V,V} 

{v,v,/} 


CSP dichotorny conjecture! P or NP-complete. 


{3,A,/} 


{v,v,=} 


Trivial if D = 1; in P if D = 2 and ^ is affine or 
bijunctive; and, NP-complete otherwise. 


{3,V,A} 
{3,V,A,=} 


{3,V,V} 
{3,V,V,/} 


a QCSP trichotomy should be conjectured: P, NP-complete, 
or Pspace-complete. 


{3,V,A,/} 


{3,V,V,=} 


Trivial if D = 1; in P if D = 2 and Q is affine or 
bijunctive; and, Pspace-complete otherwise. 


{V,3,A,V} 


Positive equality free tetrachotomy: P, NP-complete, co-NP- 
complete or Pspace-complete 


{-,3,V,A,V} 


Trivial when Q contains only trivial relations (empty or all tu- 
ples, and Pspace-complete otherwise. 


{V,3,A,V,=} {V,3,A,V,/} 
{-,3,V,A,V,=} 


Trivial when D = 1, Pspace-complete otherwise. 



Table 1 : Complexity of the model checking according to the model for syntactic fragments of FO (L 
stands for logarithmic space, P for polynomial time, NP for non-deterministic polynomial time, co-NP 
for its dual and Pspace for polynomial space). 



is the main contribution of this paper. We undertook the study of the complexity of the model check- 
ing of {3,V, A, V}-FO through the algebraic method that has been so fruitful in the study of the CSP and 
QCSP lSch78|JCG97|Bul06„BBC+09aChe08 |. To this end, we defined surjecdve hyper-endomorphisms 
and used them to define a Galois connection that characterises definability under {3, V, A, V}-FO and 
prove that it suffices to study the complexity of problems associated with the closed sets of the as- 
sociated lattice, the so-called down-closed monoids of unary surjecdve hyper-operations (DSM for 
short) |MM12a'l. Unlike the case of CSP where the corresponding lattice, the so-called clone lattice, 
is infinite and essentially uncharted when the domain size exceeds two, our lattice of DSMs is finite for 
any fixed domain. This has meant that we were able to compute the lattice for modest domain sizes, 
or charter parts relevant to our classification project, whether by hand for a domain of up to three el- 
ements [MM09I, or using a computer for up to four elements I^MMlOj]. These papers culminate in a 
full classification - a tetrachotomy - as ^ ranges over structures with up to four elements domains. 
Specifically, the problems {3, V, A, V}-FO(i^) are either in L, are NP-complete, are co-NP-complete or 
are Pspace-complete. It is a pleasing consequence of our algebraic approach that we can give a quite 
simple explanation to the delineation of our subclasses. A drop in complexity arises precisely when we 
may relativise w.l.o.g. all quantifiers of one type to a single domain element: for example, all existen- 
tial quantifiers may be fixed to a special domain element e, resulting in a natural complexity drop from 
Pspace to co-NP. Moreover, for membership of L, NP and co-NP, it is proved in rMM12a| that it is suf- 
ficient that ^ has certain special surjective hyper-endomorphisms. For our previous example, we would 
have a surjective hyper-operation, i.e. a function / from D to the power set of D, such that e £ f{d) 
for any element d of D, which is a surjective hyper-endomorphism of Intuitively, a winning strategy 
for the existential player for some input sentence (p may be transformed through "application of /" into 
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a winning strategy where any existential variable is played on the constant e. The converse, that it is 
necessary to have these special surjective hyper-endomorphisms, is more subtle and was initially only 
an indirect consequence of our exploration of the lattice of DSMs. We settled this converse direction 
and the tetrachotomy for any domain size via the introduction of the novel notion of U-X-coxt |MM11 1 
which is the analog for {3, V, A, V}-FO of the core, so useful in the case of {3, A}-FO and CSP. 

The well-known notion of the core of & may be seen as the minimal induced substructure i^' C ^ 
such that S^' and & agree on all primitive positive sentences. Equivalently, the domain D' of is 
minimal such that any primitive positive sentence is true on & iff it is true on & with all (existential) 
quantifiers relativised to D' . Cores are minimal structures in their equivalence classes, given by the 
equivalence relation of satisfying the same primitive positive sentences. Cores are very robust, for in- 
stance, being unique up to isomorphism, and sitting as induced substructures in all other structures in 
their equivalence class. A similar notion to core exists for the QCSP, but it is not nearly so robust (they 
need no longer be uniquely minimal in size nor sit as an induced substructure in other structures in their 
equivalence class |MM12b |). For the problems {3, V, A, V}-FO(^), a notion of core returns, and it is 
once again robust. The U -X-core of & consists of a minimal substructure induced by the union UVJX of 
two minimal sets U and X of D such that a positive equality-free sentence is true on & iff it is true on & 
with the universal quantifiers relativised to U and the existential quantifiers relativised to X. Analysing 
U -X-cores gives us the necessary converse alluded to in the previous paragraph. In the Pspace-complete 
case, some completion of the U-X-coxq is either fundamentally very simple and can be classified as in 
a two-element domain, known from | MM09| , or it is a generalisation of one of the four-element cases 
from [MMIOJ . For the NP-complete and co-NP-complete cases, some completion of the U-X-cor& is 
fundamentally very simple and can be classified as an easy generalisation of a three-element domain. 

We are able therefore to give the delineation of our tetrachotomy by two equivalent means. Firstly, 
by the presence or absence of certain special surjective hyper-endomorphisms, the so-called A-shops 
and E-shops (in our running example above / is an E-shop, its dual i.e. a surjective hyper-operation g 
such that there exists a constant u such that g(u) = D would be an A-shop). Secondly, by the existence 
or not of trivial sets for the relativisation of universal and existential quantifiers (see Table [2]l. Thus, 
{3, V, A, V}-FO(^) is in L iff ^ has both an A-shop and an E-shop for surjective hyper-endomorphism, 
iff there exist singleton sets U and X such that a sentence of positive equality-free logic is true on & 
exactly when it is true on ^ with the universal quantifiers and existential quantifiers relativised to U 
and X, respectively. Otherwise, and in a similar vein, {3,V, A,V}-FO(^) is NP-complete (resp., co- 
NP-complete) if it has an A-shop (resp.. E-shop) for a surjective hyper-endomorphism, iff there exists a 
singleton set U (resp., X) such that a sentence of positive equality-free logic is true on & exactly when it 
is true on & with the universal quantifiers relativised to U (resp., the existential quantifiers relativised to 
X). In all remaining cases, {3, V, A, V}-FO(^) is Pspace-complete, and & has neither an A-shop nor an 
E-shop as a surjective hyper-endomorphism, and there are no trivial sets U nor X affording the required 
relativisation properties. 

The paper is organised as follows. In Section [2j we prove preliminary results, in Section [3] we 
classify the complexity of the model checking problem for all fragments (other than those corresponding 
to CSP and QCSP) but {3, V, A, V}-FO. In Section|4j we classify the complexity of the model checking 

for {3,V,A,V}-F0. 

In more detail, in § |2.2[ we present our methodology to tackle systematically the complexity of 
the model checking problem, discuss duality and the fragments it will suffice to classify. In § |2.3[ we 



recall the notion of containment, equivalence and cores and extend it abstractly to any fragment J^. 



In § 2.5 we introduce the notion of hyper-operations and hyper-morphisms which arise naturally in 



the context of equality-free fragments. In § 2.6 we investigate containment, equivalence and core for 



{3, V, A, V, -i}-FO. In § 2.7 we characterise containment for {3,V, A,V}-FO; in § 2.8 we introduce 



the notion of a JJ-X-cove,; highlight the link with relativisation in §|2.9t prove some basic properties of 
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Tetrachotomy for {3, V, A, V}-FO(^) 


Case 


Complexity 


A -shop 


E'-shop 


U -X-core 


Relativises into 


Dual 


I 


L 


yes 


yes 




U 


= 1, 


X 


= 1 


{A,V}-FO 


I 


II 


NP-complete 


yes 


no 




U 


= 1, 


X 


>2 


{3,A,V}-F0 


III 


III 


co-NP-complete 


no 


yes 




U 


>2, 


X 


= 1 


{V,V,A}-FO 


II 


IV 


Pspace-complete 


no 


no 




U 


>2, 


X 


>2 


{3,V,A,V}-F0 


IV 



Table 2: Reformulations of the tetrachotomy {U and X denote the subsets of the domain to which uni- 
versal and existential variables relativise, respectively; the relativisation into a weaker logical fragment 
allows up to two constants). 



surjective hyper-endomorphisms of a U-X-cove, in ■ 



2.10 and, shows that it is unique up to isomorphism 



m 



In §|3j we start by recalling Schaefer's theorem for Boolean CSP, and its analog for Boolean QCSP. 
In § |3.1[ we deal with trivial fragments whose model checking problem is always in L. In 3.2 



we turn 

our attention to fragments whose complexity is trivial if the structure has a one element ^-core and is 
hard otherwise. In particular, we recall a Galois connection using hyper-endomorphisms to classify the 
fragment {3, A, V}-FO following the guidelines given by Borner |B6r08 1 regarding Galois connections. 
In 3.3 we classify the fragments whose complexity can be deduced from the Boolean case. 

In Section 41 we deal with {3, V, A, V}-FO. In § 4.1 we recall the Galois connection using surjective 
hyper-endomorphisms. In § |4.2[ we recall how the Boolean case can be classified using the lattice 
associated with this Galois connection. In general, the upper bound of our tetrachotomy is a direct 
consequence of the characterisation of U-X-core. in terms of relativisation, and we only need to deal 
with the lower bounds which we do in full generality in § 4.3 In particular, in § 4.3. 1[ we characterise 
in some detail the DSM of a U-X-core., showing that it is of a very restricted form, which allows us to 
prove hardness in a generic way in subsequent sections. Finally, In § 4.4 we investigate the complexity of 
the meta-problem: given a finite structure what is the complexity of evaluating positive equality-free 
sentences of FO over We establish that the meta-problem is NP-hard, even for a fixed and finite 
signature. 

The present paper represents the full version of the conference reports |Mar08 MM 10 MMll | and 
a part of |MM12b |; |MM1 1 1 itself supersedes a series of papers begun with |MM09 1. Unless otherwise 
stated, all results appear here for the first time (outside of conference pubhcations). 



2 Preliminaries 
2.1 Basic Definitions 

Unless otherwise stated, we shall work with finite relational structures that share the same finite relational 
signature a. Let & be such a structure. We will denote its domain by D. We denote the size of such a 
set D by \D\. The complement ^ of a structure ^ consists of relations that are exactly the set-theoretic 
complements of those in S^. I.e., for an a-ary R, := D" \ R^. For graphs this leads to a slightly 
non-standard notion of complement, as it includes self-loops. 

A homomorphism (resp. full homomorphism) from a structure ^ to a structure is a function 
h : D ^ E that preserves (resp. preserves fully) the relations of ^, i.e. for all a,-ary relations /?,-, and 
for all . . . ,Xai G D, Ri(xi, . . . G & implies Ri{h{xi), . . . ,h(xaj)) G <^ (resp. Ri(x\, . . . ,Xaj ) G ^ iff 
Ri{h{x\),. . . ,h{xa^)) G (f). ^ and (o are homomorphically equivalent if there are homomorphisms both 



5 



On the complexity of the model checking problem 



F.R. Madelaine and B.D. Martin 



from ^ to (f and from S to 

Let ^ be a fragment of FO. Let be a fixed structure. The decision problem ^{^) has: 

• Input: a sentence 9 of . 

• Question: does Qi |= (p? 

2.2 Methodology 

In this paper, we will be concerned with syntactic fragments ^ of FO defined by allowing or disallowing 
symbols from {3,V, A, V,7^,=,-i}. Given any sentence 9 in ^ , we may compute in logarithmic space 
an equivalent sentence (p' in prenex normal form, with negation pushed inwards at the atomic level. 
Since we will not be concerned with complexities beneath L, we assume hereafter that all inputs are in 
this form. 

In general Pspace membership of FO(i^) follows by a simple evaluation procedure inward through 
the quantifiers. Similarly, the expression complexity of the existential fragment {3, A, V, 7^, =}-F0 is at 
most NP; and, that of its dual fragment {V, V, A, =, 7^}-F0 is at most co-NP (in both cases, we may even 



allow atomic negation) | Var82|. We introduce formally below this principle of duality. 

Let .if be a syntactic fragment of FO defined by allowing or disallowing symbols from {3, V, A, V, / 
, =}. We denote by ^ its dual fragment by de Morgan's law: A is dual to V, 3 to V and = to 

Proposition 1. Let ^ he a syntactic fragment of FO defined by allowing or disallowing symbols from 
{3, V, A, V, 7^, =}. The problem ^{Sl) belongs to a complexity class C if and only if the problem J£{3!^ 
belongs to the dual complexity class co-C. 

Proof. For any sentence 9 in ^ , we may rewrite its negation by pushing the negation inwards until 
all atoms appear negatively, denoting the sentence hence obtained by i/A (which is logically equivalent to 
-i<p). Next, we replace every occurrence of a negated relational symbol -■/? by R to obtain a sentence of 
^ which we denote by The following chain of equivalences holds 

^ 1= ,p ^ ^ ^ ^ ^ 1= ^ Qy^^f ^^y^-^. 

Clearly, 9 can be constructed in logspace from (p and the result follows. □ 

We will use this principle of duality to only classify one fragment or its dual, for example we will 
study {3, A}-F0 and ignore its dual {V, V}-FO. We will also use this principle to classify the self-dual 
fragment {3,V, A, V}-FO. 

We assume at least one quantifier and one binary connective (weaker fragments being trivial). By the 
duality principle, we may consider only purely existential fragments, or fragments with both quantifiers. 
Regarding connectives, we have three possibilities: purely disjunctive fragments, purely conjunctive 
fragments and fragments with both connectives. Regarding equality and disequality, we should have the 
four possible subsets of {=, but it will become clear that cases with both follow the same complexity 
delineation as the case with / only. Moreover, for fragments with both quantifiers, we may use the 
duality principle between {3, V, A} and {V, 3, V} to simplify our task. This means that we would need to 
consider 3x3 positive existential fragments and 2x3 positive fragments with both quantifiers. Actually, 
we can decrease this last count by one, due to the duality between {3,V, A,V,7^}-F0 and {3,V, A,V,= 
}-F0. Regarding fragments with since we necessarily have both connectives and both quantifiers, 
we only have to consider two fragments: FO and {3,V, A,V,-i}-FO. However, we shall see that the 
complexity of FO agrees with that of {3, V, A, V, /}-F0 (and its dual {3, V, A, V, =}-F0). 

This makes a grand total of 15 fragments to classify, which are listed below; the fragments marked 
with a correspond to the CSP and QCSP and are still open. We will settle all other listed fragments. 

The 15 relevant fragments can be organised broadly in the following four classes. 
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First Class This consists of the following trivial fragments: for such a fragment ^ , the problem ^(^) 
is trivial (in L) for any structure Qi. 



{3, V}-FO (see Proposition 30 1 



{3, V, =}-F0 (see Proposition 30 1 



{3, V, /}-F0 (see Proposition 30 1 



Second Class This consists of the following fragments which exhibit a simple dichotomy: for such a 
fragment ^ , the problem ^{&) is trivial (in L) when the ^-core (defined in the next section) of & has 
one element and hard otherwise (NP-complete for existential fragments, Pspace-complete for fragments 
that allow both quantifiers). For this class, tractability amounts to the relativisation of all quantifiers to 
some constant. 



{3, A, V}-FO, {3, A, V, =}-F0 (see Proposition 33 ) 



{3, A, V, /}-F0 (see Proposition 37 ) 



{3,V,A,V,/}-F0 (see Proposition 31 ) 



{3, V, A, V,-.}-FO (see Proposition 32 ) 



Third Class This exhibits more richness complexity-wise, tractability can not be explained simply by 
^-core size and relativisation of quantifiers. 



• {3,A,/}-F0 (see Proposition |39]) 

• {3,V,A,/}-F0 (see Proposition |40l) 

★ {3,A}-F0, {3,A,=}-F0 

★ {3,V,A}-F0, {3,V,A,=}-F0 

Fourth Class The last class consists of a single fragment and is rich complexity-wise, though we will 
see that a drop in complexity is always witnessed by relativisation of quantifiers. 



{3,V,A,V}-F0 (see Theorem 41 ) 



2.3 =Sf -Containment and =Sf-Core 

It is well known that conjunctive query containment is characterised by the presence of homomor- 
phism between the corresponding canonical databases (this goes back to Chandra and Merlin | |CM77 1, 



see also |GKL"^07 chapter 6]). For exactly the same reason, a similar result holds for {3,A}-F0- 



containment. We state and prove this result for pedagogical reasons, before moving on to other frag- 



ments. The results in this section (§ 2.3 1 relating to existential fragments are essentially well known. 
Let us fix some notation first. Given a primitive positive sentence (p in {3, A}-FO, we denote by 
its canonical database, that is the structure with domain the variables of (p and whose tuples are 
precisely those that are atoms of (p. In the other direction, given a finite structure i/, we write (p^ for 
the so-called canonical conjunctive quer^of si , the quantifier-free formula that is the conjunction of 



Most authors consider the canonical query to be the sentence which is the existential quantification of (f)^/. 
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Fragment ^ 


^-containment 


^-equivalence 


^-core 


{3,A}-F0 








{3,A,=}-F0 
{3,A,V}-F0 
{3,A,V,^}-F0 


homomorphism 


homomorphic 
equivalence 


(classical) core 


{3,V,A,V}-F0 


surjective hyper- 
morphism 


surjective hyper- 
morphism equi- 
valence 


U -X-core 


{3,V,A,V,-}-F0 


Full surjective 
hyper-morphism 


Full surjective 
hyper-morphism 


quotient by ^ 


contains {3,A,7^}-FO 








or 


isomorphism 


isomorphism 


each structure 


contains {V, V,=}-FO 









Table 3: The various notions of containment, equivalence and core for syntactic fragments of FO. 



the positive facts of si , where the variables vi, . . . ,V|^| correspond to the elements ai, . . . ,a|yi| of si . It 
is well known that Q^, is homomorphic to a structure if, and only if, si \= Moreover, we now 
may define a winning strategy for 3 in the Hintikka (^2/, (p)-game to be precisely the evaluation of the 
variables given by a homomorphism from to si . Note also that si is isomorphic to the canonical 
database of 3vi3v2 . . ■v\a\^s2^- 

Theorem 2. Let si and =^ he two structures. The following are equivalent. 

(i) For every sentence (JO in {3, A}-FO, if si |= (p then ^ |= <p. 

(ii) Tiiere exists a homomorphism from si to SS. 

r--\ ^\ {3,A}-FO , {3,A}-FO -, -, 

(ill) where (p^' ' := 3vi3v2 . . . V|^| (p^. 

Proof. As we observed above, a homomorphism corresponds to a winning strategy in the [si, (p)-game 
and (ii) and (iii) are equivalent. 

Clearly, (i) impUes (iii) since si \= 3vi3v2 ...v\A\(p^. 

We now prove that (ii) implies (i). Let /j be a homomorphism from si to ^. If si \= (p, then there is 
a homomorphism g from to si. By composition, ho g is a. homomorphism from to In other 
words, hog is a. winning strategy for 3 in the <p)-game. □ 

Definition 3. Let si and ^ be two structures. We say that si is ^-contained in if, and only if, for any 
(p in , si \= (p implies |= (p. We say that si and SS are .if -equivalent if, and only if, for any cp in 
si \= (p 4^ ^ \= (p. If ^ is a minimal structure w.rt. domain size such that and si are 5£ -equivalent, 
then we say that is an .if -core of si . 

The {3, A}-FO-core is unique up to isomorphism and is better known as the core. We proceed to 
characterise notions of containment, equivalence and core for other fragments of FO, which we will 
use to study the complexity of the associated model checking problems. The results of this section are 
summarised in Table (3] 

Proposition 4. Let si and ^ he two relational structures. The following are equivalent. 
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(i) There is a homomorphism from £^ to SS. 

(ii) is {3, A}-FO-contained in 

(iii) £/ is {3, A, =}-FO-contained in 

(iv) is {3, A, \/}-FO-contained in SS. 

(v) {3, A, V, =}-FO-contained in SS. 

Proof. The equivalence of (i) and (ii) are stated in Theorem|2]and are equivalent to |= 3vi3v2 . . <jV, 
a sentence of {3, A}-FO. This takes care of the implications from (v), (iv) and (iii) to (i). Trivially (v) 
implies both (iv) and (iii). 

It suffices to prove (i) implies (v). As in the proof of Theorem [2| it can be easily checked that 
a homomorphism can be applied to a winning strategy for 3 in the (^2/, (p)-game to obtain a winning 
strategy for 3 in the (^,<p)-game. To see this, write the quantifier-free part i/A of <p in conjunctive 
normal form as a disjunction of conjunction-of-positive-atoms i//,. We may even propagate equality out 
by substitution such that each i//, is equality-free (if some y/, contained no extensional symbol other than 
equality, the sentence 9 would trivially holds on any structure as we only ever consider structures with at 
least one element). A winning strategy in the (=2/, <p)-game corresponds to a homomorphism from some 

to .s^ . By composition with the homomorphism from si to we get a homomorphism from 3)^. 
to i.e. a winning strategy in the (p)-game as required. □ 

Corollary 5. Let £/ and ^ be two relational structures. The following are equivalent. 

(i) £^ and 3S are homomorphically equivalent. 

(ii) and 3S have isomorphic cores. 

(iii) is {3, f\]-FO-equivalent to SS. 

(iv) is {3, A, =\-FO-equivalent to ^. 

(v) J2/ is {3, A, \J^-FO-equivalent to SS. 

(vi) si is {3, A, V, =]-FO-equivalent to SS. 
We now move on to fragments containing {3, A, 7^}. 

Proposition 6. Let si and ^ he two structures. The following are equivalent. 

(i) For every sentence (p in {3, A, 7^}-F0, if s/ \= (p then ^ |= (p. 

(ii) There exists an injective homomorphism from s/ to 

{\\\) m\=(py where (^1/ := 3vi . . . V|a|<|V A Ai</<;<|a| v; ^ v^-. 

Proof. Similar to Theorem [2] 

Corollary 7. Let ^ he a fragment of FO such that ^ or its dual ^ contains {3, A, 7^}-F0 
he two structures. The following are equivalent. 

(i) si and ^ are isomorphic. 

(ii) si is ££ -equivalent to 

Proof. For the case when ££ contains {3, A, /}-F0, the result follows from the previous proposition and 
the fact that we deal with finite structures only. 

For the case when ££ contains {3, A,/}-FO, we apply the duality principle and the previous case, 
and equivalently si and are isomorphic. This is in turn equivalent to si being isomorphic to SS. □ 
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2.4 Hintikka Games 

Before moving on to the equality-free fragments {3, V, A, V}-FO and {3, V, A, V, -i}-FO, let us recall 
first basic definitions and notations regarding Hintikka Games. Let <p be a sentence of FO in prenex form 
with all negations pushed to the atomic level. A strategy for 3 in the (Hintikka) (iz/, <p)-game is a set 
of mappings {Ox : '3x' G <p} with one mapping Ox for each existentially quantified variable x of (p. The 
mapping Ox ranges over the domain A of £/; and, its domain is the set of functions from Yx to A, where 
Yx denotes the universally quantified variables of (p preceding x. 

We say that {Ox : '3;c' G cp} is winning if for any assignment 7i of the universally quantified variables 
of (p to A, when each existentially quantified variable x is set according to Ox applied to 71 , then the 
quantifier-free part i/a of (p is satisfied under this overall assignment h. When i/a is a conjunction of 
positive atoms, this amounts to h being a homomorphism from to £/. 

2.5 Hyper-morphisms 

For the equality-free fragments {3,V, A,V,-i}-FO and {3,V, A,V}-FO, the correct concept to transfer 
winning strategies involves unary hyper-operations, that is functions to the power-set. 

A hyper-operation f from a set A to a set B is a function from A to the power-set of B. For a subset 
S of A, we will define its image /(A) under the hyper-operation / as U^gs When we wish to stress 
that an element may be sent to 0, we speak of a partial hyper-operation; and otherwise we assume that / 
is total, that is for any a in A, f{a) ^ 0. We say that / is surjective whenever /(A) = B. The inverse of a 
(total) hyper-operation / from A to B, denoted by /^^ is the partial hyper-operation from B to A defined 
for any in B as (Zj) := {a G A | G /(a)}- We call an element of {b) an antecedent of b under /. 
Let / be a hyper-operation from A to B and g a hyper-operation from B to C. The hyper-operation go f 
is defined naturally as go f{x) := g(/(x)) (recall that f{x) is a set). 

When / is a (total) surjective hyper-operation from A to A, we say that / is a shop of A. Note that 
the inverse of a shop is a shop and that the composition of two shops is a also a shop. Observing further 
that shop composition is associative and that the identity shop (which sends an element x of A to the 
singleton {x}) is the identity with respect to composition, we may consider the monoid generated by a 
set of shops. A shop / is a sub-shop of a shop g whenever, for every x in A, /(x) C g{x). In our context, 
we will be interested in a particular monoid which will be closed further under sub-shops, a so-called 
down-shop-monoid (DSM).|^We denote by {F)dsm the DSM generated by a set F of shops. 

Let / be a shop of A. When for a subset U of A we have f{U) = A, we say that / is U -surjective. 
Observing that the totality of / may be rephrased as /^^ (A) = A, we say more generally that / is X-total 
for a subset X of A whenever f^\X) = A. Note that for shops ?7-surjectivity and X-totality are dual 
to one another, that is the inverse of a ?7-surjective shop is an X-total shop with X = U and vice versa. 
Somewhat abusing terminology, and when it does not cause confusion, we will drop the word surjective 
and by ?7- or U'-shop we will mean a U- or ?7'-surjective shop. Similarly, we will speak of an X- or 
X'-shop in the total case and of a ?7-X-shop in the case of a shop that is both ?7-surjective and X-total. 
Suitable compositions of ?7-shops and X-shops preserve these properties. 

Lemma 8. Let f and g be two shops. 

(i) If f is a U -shop then go f is a U -shop. 

(ii) If g is a X-shop then go f is a X-shop. 

(iii) If both f is aU -shop and g is a X-shop then go f is aU -X-shop. 



^The "down" comes from down-closure, here under sub-shops; a nomenclature inherited from IBorOO 
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(iv) If both f and g are U -X -shops then go f is aU -X-shop. 

(v) The iterate of a U -X-shop is a U -X-shop. 

Proof. We prove (i). Since f{U) = A, we have g{f{U)) = g{A). By surjectivity of g, we know that 
g{A) =A. It follows that g{f{U)) =A and we are done, (ii) is dual to (i), and (iii) follows directly from 
(i) and (ii). (iv) is a restriction of (iii) and is only stated here as we shall use it often, (v) follows by 
induction on the order of iteration using (iv). □ 

A hyper-morphism f from a structure to a structure ^ is a hyper-operation from A to 5 that 
satisfies the following property. 

• (preserving) if R{ai,. ..,ai) £ ^ thenR{bi,. . .,bi) £ ^, for all b\ £ /(ai), . .■,bi£ f{ai). 

When and ^ are the same structure, we speak of a hyper-endomorphism. We say that / is full if 
moreover 

• (fuUness) /?(ai, . . . G ^ \SiR{bu. . .,bi) e SS, for all bx G /(ai), . . . G /(a,). 

Note that the inverse of a full surjective hyper-morphism is also a full surjective hyper-morphism. 

2.6 Containment and Core for { 3 , V, A , V , -i } -FO 

We now turn our attention to {3,V, A, V,-i}-FO. The proofs of the necessary characterisations are some- 
what laboured, but will prepare us well for the forthcoming discussion on {3, V, A, V}-FO. 

Lemma 9. Let and ^ be two structures such that there is a full surjective hyper-morphism from 
to SS. Then, for every sentence <p in {3, V, A,V,-i}-FO, if |= 9 then ^ \= (p. 

Proof. Let /i be a full surjective hyper-morphism from £/ to and <p be a sentence of {3, V, A, V, -i}-FO 
such that £/ 1= (p. We fix an arbitrary linear order over A and write mmh~^{b) to denote the smallest 
antecedent of in A under h. 

Let {<Jx : G <p} be a winning strategy in the <p)-game. We construct a strategy {o^ : G <p} 
in the <p)-game as follows. Let ;re : — ^ ^ be an assignment to the universal variables Yx preceding 
an existential variable x in (p, we select for CJ^(7r) an arbitrary element of h{a{7lA)) where Ka - Yx ^ A 
is an assignment such that for any universal variable y preceding x, we have nA{y) := minh~^{7rB{y)). 
This strategy is well defined since h is surjective (which means that tta is well defined) and total (which 
means that h{(7{KA)) 7^ 0). Note moreover that using min in the definition of TTa means that a branch in 
the tree of the game on ^ will correspond to a branch in the tree of the game on £/. It remains to prove 
that {<7^ : '3x' G (p} is winning. We will see that it follows from the fact that h is full and preserving. 

We assume that negations have been pushed to the atomic level and write the quantifier-free part Y 
of <p in disjunctive normal form as a disjunction of conjunctions-of- atoms If Vi contradictory 
positive and negative atoms (as in E{x,y) A -'E{x,y)) then we may discard the sentence Yi false. 
Moreover, for each pair of atoms R{vi,V2,...,Vr) and -i/?(vi, V2, . . . , v^) (induced by the choice of a 
relational symbol R and the choice of r variables vi , V2, . . . , occuring in Xjfi) such that neither is present 
in 1//,, we may replace Xjfi by the logically equivalent (i//; A /?(vi , V2, . . . , v^)) V ( V^,- A -i/?(vi , V2, . . . , v^)) . 
After this completion process, note that every conjunction of atoms Xjfi corresponds naturally to a structure 
^y,. (take only the positive part of Yi which is now maximal). 

Assume first that Y is disjunction-free. The winning condition of the <p)-game can be recast as a 
full homomorphism from to Composing with h the full homomorphism from to (induced 
by the sequence of compatible assignments Ka to the universal variables and the strategy {Ox : G 
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<p}), we get a full hyper-morphism from to SS. The map from the domain of to induced 
by the sequence of assignments tzb and the strategy {a^ : '3;t:' G <p} is a range restriction of this full 
hyper-morphism and is therefore a full homomorphism (we identify hyper-morphism to singletons with 
homomorphisms). In general when the quantifier-free part of <p has several disjuncts i//,, most likely 
after the completion process of the previous paragraph, the winning condition can be recast as a full 
homomorphism from some ^y,.. The above argument applies and the result follows. □ 

We shall see that the converse of Lemma [9] holds. Consequently, it turns out that containment and 
equivalence coincide for {3, V, A, V, -i}-FO, since the inverse of a full surjective hyper-morphism is a 
full surjective hyper-morphism. 

For {3,V, A, V, -i}-FO, we define an equivalence relation ~ over the structure elements in the spirit of 
the Leibnitz-rule for equality. For propositions P and Q, \eXP ^ Q be an abbreviation for (PA 2) V (-iP A 
-■2). For the sake of clarity, we deal with the case of digraphs first and write ;c ~ j as an abbreviation for 
Vz(£'(x,z) ^E(y^z)) r\ {E{z,x) ^ E{z,y)). It is straightforward to verify that ~ induces an equivalence 
relation over the vertices (which we denote also by ~). In general, for each r-ary symbol R, let Yr stands 
for 



[R{x,Zl,...,Zr-l) ^ R(y,Zu.-.,Zr-l)) A (P(zi,X,Z2, • • • ,Zr-l) ^ Rizi,y,Z2,---,Zr-l)) 

A ... A (P(Z1,Z2, • • • ,Zr-Ux) O P(Z1,Z2, • • • ,Zr-Uy)) ■ 

We write ;c ~ y for Argct ^^i ,Z2, ■ ■ ■ ,Zr-iVR- 

We write for the quotient structure defined in the natural way. Note that there is a full surjective 
homomorphism from to As observed earlier, its inverse (viewing the homomorphism as an 

hyper-morphism) is a full surjective hyper-morphism from to £/. Thus, it follows from Lemma |9] 
that £/ and £//^ are {3, V, A, V, -ij-FO-equivalent. 

Let (p^ denotes the (quantifier-free) canonical conjunctive query of £/ (denoted earlier as <jv) and 
(p^ denotes the similar sentence which lists the negative atoms of £/ instead of the positive atoms. 

Proposition 10. Let £/ and he two structures. The following are equivalent. 

(i) For every sentence (p in {3, V, A, V, -i}-FO, if |= (p then 3^ \= (p. 

(ii) There exists a full surjective hyper-morphism from s/ to 
(ill) 3& \= (pj;^ ' where 

9^ :=3vi3v2...V|4|<p^A<p^AVw y w~v,-. 

1<!<|A| 



(iv) for every sentence (p in {3, V, A, V, -i}-FO, £/ \= (p iff ^ \= (p. 

(v) j# /~ and are isomorphic. 

Proof. The implication (i) to (iii) is clear since by construction £/ models the canonical sentence cp^'^'^'"^ 

We prove that (iii) implies (ii). Assume that ^ |= (p^''^'^''^'^^ We construct a full and total 
surjective hyper-morphism h as follows. Let b\,b2, - ■ ■ be witnesses in B for vi,V2, . . . ,V|^|. We set 
h{ai) B hi for 1 < / < |A| (totality). For each b in B, we set the universal variable w to ^ and pick some j 
such that w ~ vj holds and set h{aj) B b (surjectivity). By construction, h is preserving and full. 

The implication (ii) to (i) is proved as Lemma |9] 
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The equivalence of (i), (ii), (iii) with (iv) follows from our earlier observation that the inverse of 
a full surjective hyper-morphism / from jz/ to =^ is a full surjective hyper-morphism from SS to . 

To see that (v) implies (ii), compose the quotient map from si to (which is a full surjective 

homomorphism) with the inverse of the quotient map from to i^/^ (which is a full surjective hyper- 
morphism). 

For the direction (ii) to (v), the natural quotient //~ of a full surjective hyper-morphism / from si 
to =^ is a full surjective homomorphism. Since we deal with finite structures, it is an isomorphism and 
we are done. □ 

Note that no smaller structure can be {3,V, A,V,-i}-FO-equivalent to si' := Indeed, a full 

surjective hyper-morphism / from a smaller structure to .si' would have to satisfy {a']^,a'2\ ^ f{b) 
for some b inB and some distinct a'^,a'2 in A'. But this would imply that a'^ ~ a'2 which is not possible. 
Moreover, any structure that is {3, V, A, V, -i}-FO-equivalent and of the same size as si' will be isomor- 
phic (a full surjective hyper-morphism must induce an isomorphism by triviality of ~ over .s^'). Thus, 
si /~ is the (up to isomorphism unique) {3,V, A,V,-i}-FO-core of si. 



2.7 Containment for {3, V, A, V}-FO 

Lemma 11. Let si and he two structures such that there is a surjective hyper-morphism from si to 
Then, for every sentence (p in {3, V, A, V}-FO, if si |= <p then ^ |= (p. 

Proof. The proof is exactly the same as that of Lemma [9j except that we no longer need to preserve 
atomic negation, and may drop the assumption of fullness. □ 

We extend the notion of canonical conjunctive query of a structure si . Given a tuple of (not necessar- 
ily distinct) elements r := (ri, . . . G A', define the quantifier-free formula ^/(r) (vi , • • • ,v/) to be the 
conjunction of the positive facts of r, where the variables vi , . . . , v/ correspond to the elements ri , . . . , r/. 
That is, R{vxi ,vx.) appears as an atom in <p_<3/(r) iff ^(''Ai ^ ■ - ^^x) holds in si . When r enumerates the 
elements of the structure si , this definition coincides with the usual definition of canonical conjunctive 
query. Note also that in this case there is a full homomorphism from the canonical database ^<fl^,y^^■^ to si 
given by the map v^. 1— )• r,-. 



Definition 12 ( |MM12a|). Let .si he a structure and m > 0. Lef r he an enumeration of the elements of 
si. 



:=3vi,...,V|A|<|V(r)(vi,...,V|A|)AVwi,...,W„ Y <?V(r,t)(v,w). 



Observe that .si \= 6^^^'^^^ Indeed, we may take as witness for the variables v the corresponding 
enumeration r of the elements of si ; and, for any assignment t G A'" to the universal variables w, it is 
clear that si \= <jV(r,t)(r,t) holds. 

Lemma 13. Let si and be two structures. If ."SS \= ^^^f^j^^^ then there is a surjective hyper- 
morphism from si to SS. 

Proof. Let h' :=b'^, . . . ,b'^j^^ be witnesses for vi, . . . ,v\a\. Assume that an enumeration h := bi,b2, ■ ■ ■ ,b\g\ 
of the elements of ^ is chosen for the universal variables w\,. . - w^ag^. Let t G A'" be the witness s.t. 

^ N <IV(r)(b') A(p.^(r,t)(b',b). 

Let / be the map from the domain of si to the power set of that of ^ which is the union of the 
following two partial hyper-operations h and g (i.e. /(a,) := h{ai) VJ g{ai) for any element a,- of .si), 
which guarantee totality and surjectivity, respectively. 
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• h(ai) ■=b'f (totality.) 

• g{tj) 3 bj (surjectivity.) 

It remains to show that / is preserving. This follows from ^ |= (j)j/(r.t)(b',b). 

Let /? be a r-ary relational symbol such that /?(<3,-, , . . . ,a,,,) holds in sii . Let b'!^ G f{ai^),. . . ,b'/^ G 
/(a,.). We will show that R{b'l^ ,b'l) holds in Assume for clarity of the exposition and w.l.o.g. that 
from i\ to the image is set according to h and from to ir according to g: i.e. for \ < j <k, h{aij) = 
b'l. = b'l'. and for + 1 < j <r, there is some Ij such that ti. = Uj. and g{ti-) 3 b'/. = bi-. By definition of 
i2/ (r, t) the atom R{vi^ , . . . , v;, , w/j.^, , . • • , w^) appears in <|V(r^t) (v, w). It follows from ^ |= <?V(r_t) (b', b) 
that /? , ... , b'l ) holds in □ 

Theorem 14. Let si and be two structures. The following are equivalent. 

(i) For every sentence (p in {3, V, A, V}-FO, if s/ |= (p then |= <p. 

(ii) There exists a surjective hyper-morphism from si to SS. 

(iii) ^ 1= eiT.r^"'° 

Proof. By construction si |= so (i) implies (iii). By Lemma |Il| (ii) implies (i). By 

Lemma [T3| (iii) implies (i). □ 



2.8 A core for {3,V, A, V}-FO 

The property of a (classical) core can be rephrased in the logical context as the minimal X = A C A 
such that a primitive positive sentence <p is true on si iff it is true on si with the (existential) quantifiers 
relativised to X = A. Let us say in this case that si has X-relativisation with respect to {3, A}-FO. 

Thus, the notion of a core can be recast in the context of {3, A}-FO in a number of equivalent ways, 
as a minimal induced substructure si of si, 

(i) that satisfies the same {3, A}-FO sentences; 

(ii) that is induced by minimal X C A such that si has X-relativisation w.r.t. {3, A}-FO; or, 

(iii) that is induced by minimal X C A such that si has an endomorphism with image X. 

We are looking for a useful characterisation of the analogous concept of core for {3, V, A, V}-FO. As 
we now have both quantifiers, two sets U and X, one for each quantifier, will emerge naturally, hence 
we will call this core a U-X-core. As we shall see shortly, there are two ecjuivalent ways of defining a 
U-X-coie - one is logical, the other algebraic - as a minimal substructure si of si, induced by minimal 
U,X CA such that: 

(ii) si has \/U -3X-relativisation w.r.t. {3,V, A, V}-FO; or, 

(iii) si has a ?7-surjective X-total hyper-endomorphism. 

Recall that a surjective hyper-endomorphism f of si is U -surjective if f{U) = A and X -total if 
f-\X)=A. 

We will show that the sets U and X are unique up to isomorphism and that within a minimal induced 
substructure si, the sets U and X are uniquely determined. This will reconcile our definition of a U-X- 
core with the following natural definition, in which U and X are not explicit: 

(i) as a minimal induced substructure si of si that satisfies the same sentences of {3, V, A, V}-FO. 

In our definition of {3, V, A, V}-FO-core, we ask for a minimal structure, i.e. not necessarily an induced 
substructure. We shall see that it is equivalent to the above. 
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2.9 Relativisation 

Given a formula cp, we denote by (P[\^u/^ueu, 3x/3xex] the formula obtained from cp by relativising simul- 
taneously every universal quantifier to U and every existential quantifier to X. When we only relativise 
universal quantifiers to U, we write (P[yu/yueu]' when we only relativise existential quantifiers to X, 
we write (P[3j,/3j,ex]- 

Definition 15. Let £/ be a finite structure over a set A, and U,X be two subsets of A. We say that has 
\/U -3X -relativisation if, for all sentences (p in {3, V, A, V}-FO the following are equivalent 

(/) 1= <p 

(//) 1= <P[v„/v«ec/] 
(///) 1= (P[3xl3xeX\ 

{iv) J2^ 1= <P[V«/VMeC/.3.v/3.vGZ] 

Lemma 16. Let si be a finite structure over a set A, and U ,X be two subsets of A. If has aU- 
surjective X-total hyper-endomorphism then si has \/U -3X -relativisation. 



Proof. Note that in Definition [15} we have (///) =^ (/) =^ (//) and (///) =^ {iv) =^ (//) trivially. It suffices 
to prove that (//) =^ (/) and (/) =^ (///) to complete the proof. To do so, we will consider the well known 
Hintikka game corresponding to Case (/), called the unrelativised game hereafter; and, the relativised 
Hintikka games corresponding to the relativised formulae from Cases (//), (///) and {iv) (the relativised 
game considered being clear from context). 

Let h be a ?7-surjective X-total surjective hyper-endomorphism of The proof follows the line of 
that of Lemma nn 

((//) =^ (/)). Assume that we have a winning strategy in the universally relativised game. We produce 
a winning strategy in the unrelativised game using h. When taking the antecedent of a universal variable, 
we make sure to pick an antecedent in U which we can do by ?7-surjectivity of h. To be more precise. 



the linear order over si used in the proof of Lemma 1 1 starts with the elements of U . 



((/) =^ (///)). Assume that we have a winning strategy in the unrelativised game. We produce a 
winning strategy in the existentially relativised game using h. When taking the image of an existential 
variable, we no longer pick an arbitrary element but one in X, which we can do by X-totahty of h. □ 



Proposition 17. The following are equivalent. 

(i) jz/ has \/U -3X -relativisation. 

(ii) has \/X-3U -relativisation. 

Proof. It suffices to prove one implication. We prove (ii) implies (i). Let (p be a sentence of {3, V, A, V}-FO, 
We use the duality principle and prove that s/ \= (p <^=^ s/ \= (P[yu/\jueu]- The other cases are similar 
and are omitted. 

We follow the same notation as in Proposition [T] y is the sentence logically equivalent to -i<p with 
negation pushed at the atomic level, and Jp is the sentence obtained from y by replacing every occurrence 
of a negative atom by R. Recall the following chain of equivalence. 

s/\=(p <^ si 1= -■(-■(?)) <?=^ si 1= -.(!//■) <^ s/^Y ^ ^ 9. 
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By assumption iz/ ^ <p <^=^ si ^ <P[3«/3Hej/]- Using the above chain of equivalence backward and 
propagating the relativisation we obtain the following chain of equivalence. 

3«/3nG{7] 3m/3hgC/] J 

□ 

Lemma 18. Let si he a finite structure over a set A, and U,X be two subsets of A. If ^ has \/U-3X- 
relativisation then s/ has a U -surjective X-total hyper-endomorphism. 

Proof. Using the fact that the identity (defined as i{x) := {x} for every x in si) is a surjective hyper- 
endomorphism of si and applying Theorem 14 we derive that si |= By assumption, we 
may equivalently relativise only the existential quantifiers to X (Definition 15 (/) =^ (///)) and si \= 

^^_'\A\[3x/3xexy Proceeding as in the proof of Lemma |Il| but over this relativised sentence, we derive the 
existence of an X-total surjective hyper-operation g. 

Using Proposition 17 and working over si , we derive similarly that si has a U -total surjective hyper- 
operation. Let / be the inverse of this hyper-operation. Observe that it is a ?7 -surjective hyper-operation. 

By Lemma[8] the composition of these operations g o / is a X-total U -surjective hyper-endomorphism 
as required. □ 

Together, the two previous lemmata establish an algebraic characterisation of relativisation. 

Theorem 19. Let si be a finite structure over a set A, and U,X be two subsets of A. The following are 
equivalent. 

(i) The structure si has \/U -3X -relativisation. 

(ii) The structure si has a X-total U -surjective hyper-endomorphism. 

Corollary 20. Let si be a finite structure that has a IJ -surjective X-total hyper-endomorphism. Let si 
be the substructure of si induced by UUX. The following holds. 

(i) si and si are {3,y,A,V}-F0-equivalent. 

(ii) jz/ has \/U -3X -relativisation. 

Proof. Let / be the U -surjective X-total hyper-endomorphism of si. Its range restriction g to A = UUX 
is a surjective hyper- morphism from si to si. The inverse g^^ of g is a surjective hyper-morphism from 
si to si, by X-totality of /. Appealing to Lemma \u\ twice, once with g and once with g^^, we obtain 

The restriction of g to A is a ?7-surjective X-total hyper-endomorphism of si, and (ii) follows from 
Lemma [T6l □ 



2.10 The U-X Core 

Given a structure ^, we consider all minimal subsets X of D such that there is an X-total surjective hyper- 
endomorphism g of and all minimal subsets U such that there is a ?7-surjective hyper-endomorphism 
/ of Such sets always exist as totality and surjectivity of surjective hyper-endomorphisms mean that 
in the worst case we may choose U = X =D. Since ^ o / is a X-total U -surjective hyper-endomorphisms 
of & by Lemma [8} we may furthermore require that jimong all minimal sets satisfying the above, we 
choose a set U and a set X with U HX maximal. Let & be the substructure of ^ induced by U UX. We 
call ^ a U-X-core of ^. 
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Remark 1. Assume that there is an Xi-shop h[ and an X2-shop h2 that preserves 3) such that |Xi | > |X2|. 
We consider images ofh\o h2. For each element in Xj, pick a single element ofX\ in h\ {X2) HXi such 
that x[ G hi (X2). Let X[ denote the set of picked elements. Since \Xi \ > IX2I then hi o h2 is an X[-shop 
that preserves Si with \X[\ < \X2\. Diagrammatically, this can be written as, 

D %X2^X'i'Zhi{X2)nXi CXi Chioh2{D). 

This means that we may look for an X-shop where the set X is minimal with respect to inclusion, 
or equivalently, for a set with minimal size \X\. So, in order to find an X-shop with a minimal set \X\, 
we may proceed greedily, removing elements from D while we have an X-shop until we obtain a set X 
such that there is no X' -shop for X' C X. The dual argument applies to U -shops, and consequently to 
U -X-shops. 

This further explains why minimising U and X, and then maximising their intersection, necessarily 
leads to a minimal D:=UL)X also. Because, would we find U'UX' of smaller size, we might look within 
U' and X' for potentially smaller sets of cardinality \U\ and \X\, thus contradicting minimality. 

Note that the sets U and X are not necessarily unique. However, as we shall see later the U-X-core 



is unique up to isomorphism (see Theorem 27 1. Moreover, within S, the sets U and X are uniquely 



determined. We delay until later the proof of this second result (see Theorem 50 1. 



2.11 Uniqueness of the U -X-core 

Throughout this section, let S he a finite structure and ^ its associated DSM; i.e. ^ is the set of 
surjective hyper-endomorphisms of S. Let U and X be subsets of D such that the substructure D of ^ 
induced byD = UUXisa.U -X-core of S. We will progress through various lemmata and eventually 
derive the existence of a canonical ?7 -X-shop in ^ which will be used to prove that the ?7-X-core is 
unique up to isomorphism. Uniqueness of the U -X-core has no real bearing on our classification program 
but the canonical shop will allow us to characterise all other shops in which will be instrumental in 
the hardness proofs for {3, V, A, V}-FO(^). 

Lemma 21. Let f be a shop in For any element z in D, f{z) contains at most one element of the set 
U, that is \f{z)nU\ < 1. 

Proof. Assume for contradiction that there is some z and some distinct elements ui and U2 of U such that 
/(z) 5 {ui,U2}. Let Z3,Z4, . . . be any choice of antecedents under / of the remaining elements uj,U4,. .. 
of U (recall that / is surjective). By assumption the monoid ^ contains a ?7-shop g. Hence, go f 
would be a U'-shop with U' = {z,Z3,Z4, ■ ■ ■} since f{U') C U and g{U) = D. We get a contradiction as 
\U'\<\U\. □ 

Lemma 22. Let f be aU -shop in There exists a permutation aofU such that: for any u in U, 

(i) f{u)r\U = {a{u)};and, 

(ii) f-'{u)nu = {a-'{u)}. 



Proof. It follows from Lemma 21 that for any uinU, \f{u) r]U\ < 1 . Since f is aU -shop, every element 
in D has an antecedent in U under / and thus in particular for any u in U, \f^^{u) r\U\ > 1. Note that 
if some element of U had no image in U then as U is finite, we would have an element of U with two 
distinct images in U. Hence, for any u in U, \f{u) nU\ = I and the result follows. □ 

The dual statements concerning X-shops hold. 



17 



On the complexity of the model checking problem 



F.R. Madelaine and B.D. Martin 



Lemma 23. Let f be a shop in for any element z in D, f ' (z) contains at most one element of the 
setX, that is |/"'(z)nX| < 1. 



Proof. By duality from Lemma 21 



□ 



Lemma 24. Let f be an X-shop in There exists a permutation (5 ofX such that: for any x in X, 

(i) f{x)nX = {p{x)}; and, 

(ii) f-\x)nx = {l5-\x)}. 



Proof By duality from Lemma 22 



□ 



Lemma 25. Let f be a shop in If f is a U -X-shop then f{X) (^{U\X) =0. 

Proof. Assume for contradiction that for some xi eXandsomewi G ?7\X, we have mi Since/ 
is an X-shop, every element is an antecedent under / of some element in X, in particular every element 



X2,X3, . . . G X (different fromxi) has a unique image X2,X3, . . . G X (see Lemma 24 1. Some element of X, 
say Xi does not occur in these images. Necessarily, x\ reaches x,. Note that x, can not also belong to U as 



otherwise, x, and u\, two distinct elements of U , would be reached by xi, contradicting Lemma 21 Thus, 
we must have that x,- belongs toX\U. Let U' ■.= U and X' := X \ {x;} U {mi }. Note that f^ := fof, the 
second iterate off, is a ?7'-X'-shop with |?7'| = |?7|, |X'| = |X| and |?7'nX'| < |?7nX|. This contradicts 
our hypothesis on U and X. □ 

Proposition 26. Let ^ be a DSM over a set D and U and X be minimal subsets ofD such that: there is 
a U -shop in there is an X-shop in and UL)X is minimal. Then, there is a U -X-shop h in ^ that 
has the following properties: 

(i) foranyyinUnX, h{y)n{ULlX) = {y}; 

(ii) for any xinX\ U, h{x) n (f/ UX) = {x}; 

(iii) for any u in U\X, h{u) n {U UX) = {u} UX„, where X„ C X \ U; and, 

(iv) /j([/\X)nX = U„ef;\X^«=X\«7. 

Proof. By assumption, ^ contains a U -X-shop /. Let a and j3 be permutations of U and X, respectively. 



as in Lemmata |22] and 24 Let r be the least common multiple of the order of the permutations jS and 
a. We set h to be the rth iterate of / and we now know that h{z) B z for any element z and that /j is a 
[/-X-shop by [8] Let y inU nX, we know that h{y) 3 y. We can not have another element from U UX in 
h{y) by Lemmata [2T] and 24 This proves (i). Let x in X \ ?7, we know that h{x) 3 x. We can not have an 



element from X distinct from x in h{x) by Lemma 24 and we can not have an element from U \X in h{x 



by Lemma 25 This proves (ii). Let m in ?7 \X, we know that h{u) B u. We can not have an element from 



U distinct from u in h{u) by Lemma 22 We may have however some elements from X\U in h{u). Thus, 
there is a set C Xj, C X \ ?7 such that h{u) n {U UX) = {«} UX„. This proves (iii). By construction 
/j is a [/-shop and every element must have an antecedent in U under h. Since by the first three points, 
elements from X\U can only be reached from elements inU\X, the last point (iv) follows. □ 



Theorem 27. The U -X-core is unique up to isomorphism. 
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Proof. Let h\ be a ?7i-Xi-shop with minimal \U\\, \X\\ and \U\ VJX\\ and let h2 be a ?72-^2-shop with 
minimal \U2\, \X2\ and |?72UX2|. Hence, h\ oh2 is a/ji(X2)nXi-shop with |/ii(X2)| < |Xi|. By minimality 
of Xi, \hi{X2)\ = \X\\, and the restriction of hi to domain X2 and codomain Xi induces a surjective 
homomorphism from the substructure induced by X2 to the substructure induced by Xi. Similarly h2 
induces a surjective homomorphism in the other direction. As we work with finite structures, hi induces 
an isomorphism / from the substructure induced by Xi to the substructure induced by X2. By duality, we 
also get that hi induces an isomorphism /' from the substructure induced by i/i to the substructure induced 
by 1/2- By construction, / and /' agree on Ui HXi (necessarily to U2 nX2) and the result follows. □ 



3 Complexity classification 

We recall below some well known results concerning the complexity of Boolean CSP and QCSP which 
we will need later. For definitions and further details regarding the proof, the reader may consult the nice 
survey by Chen | ,Che09J . 



Theorem 28 ( ]Sch78[ ). Let & be a Boolean structure. Then CSP{&) (equivalently, {3, A}-FO(^)j is 
in P if, and only if all relations of Qi are simultaneously 0-valid, I -valid, Horn, dual-Horn, bijunctive or 
affine, and otherwise it is NP-complete. 

A similar result holds when universal quantifiers are added to the mix. (it was sketched in the pres- 
ence of constants | Sch78 1, then proved in the absence of constants in |CKS01 1 and | |Dal97[ ). 



Theorem 29. Let & be a Boolean structure. Then QCSP{&) (equivalently, {3,V, A}-FO(^)j is in P if 
and only if, all relations of & are simultaneously Horn, dual-Horn, bijunctive or affine, and otherwise it 
is Pspace-complete. 

Example 1. The canonical example of a relation that does not fall in any of the tractable cases is 
NAE := {0, 1}^ \ {(0,0,0), (1, 1, 1)}. Let =^nae the Boolean structure with this relation. It follows 
from the above theorems that CSP{c^]^ae) ^P-complete and that QCSP{,'^^ae) Pspace-complete. 

Example 2. For larger domains, though the classification remains open, the canonical hard problem 
is induced by the relation 7^. Let .J^n denote the clique of size n {we view an undirected graph as a 
structure with a single binary predicate E that is symmetric). For n>3, CSP{J^n) i^ ci reformulation of 
the n-colourability problem and is -complete. It is also known that for n>3 QCSP{J(fn) is Pspace- 



complete [BBC+091 



3.1 First Class 

Proposition 30. (i) When ^ has a single element, the model checking problem for FO is in L. 

(ii) The model checking problem {3, V, 7^, =}-F0 is in L. 

Proof. (i) In the case where \D\ = 1, every relation is either empty or contains all tuples (one tu- 
ple), and the quantifiers 3 and V are semantically equivalent. Hence, the problem translates to the 
Boolean Sentence Value Problem (under the substitution of and 1 for the empty and non-empty 



relations, respectively), known to be in L |Lyn77|. 



(ii) We may assume by the previous point that \D\ > 1. We only need to check if one of the atoms 
that occurs as a disjunct in the input sentence holds in i^. Since |D| > 1, a sentence with an atom 
like X = y or X y^y is always true in ^. For sentences of {3, V}-FO, the atoms may have implicit 
equality as in R{x,x,y) for a ternary predicate R: in any case, each atom may be checked in constant 
time since ^ is a fixed structure, resulting in overall logspace complexity. □ 

□ 
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3.2 Second Class 

We now move on to the largest fragments we will need to consider, which turn out to exhibit trivial 
dichotomies. 

Proposition 31. In full generality, the class of problems {3,V, A, V, 7^}-F0(i^) exhibits dichotomy: if 
\D\ = 1 then the problem is in L, otherwise it is Pspace-complete. Consequently, the fragment extended 
with = follows the same dichotomy. 

Proof. When \D\ > 2, Pspace-hardness may be proved using no extensional relation of D other than 
The formula (p.je,o\ i^^y) •= ¥^ y) simulates the edge relation of the clique and the problem 



{3,V, A}-FO(J^) better known as QCSP(^) is Pspace-complete for « > 3 | jBBC+09[ . For n = 2, we 



use a reduction from the problem QCSP(,^nae) to prove that {3, V, A, V}-FO(J^) is Pspace-complete. 
Let (p be an input for QCSP(^nae)- Let (p' be built from (p by substituting all instances of NAE{x, y,z) 
by E{x,y) V E{y,z.) V E{x,z). It is easy to see that =^nae N *P =^ N the result follows. 

Note that we have not used = in our hardness proof; and, in the case |D| = 1, we may allow = without 
affecting tractability (triviality). Thus, the fragment extended with = follows the same delineation. □ 

Proposition 32. In full generality, the class of problems {3, V, A, V, -i}-FO(^) exhibits dichotomy: if 
all relations of S> are trivial (either empty or contain all tuples) then the problem is in L, otherwise it is 
Pspace-complete. 

Proof. If all relations are trivial then has a single equivalence class. Thus, has a single element. 



By {3,V, A, V, -i}-FO-equivalence of ^ and (see Proposition 10 1, it suffices to check whether an 
input (p in {3, V, A, V, -i}-FO holds in ^/~. Since the latter has a single element, the problem is in L. 
Otherwise, the equivalence relation ~ has at least 2 equivalence classes since is non trivial. We 



may now follow the same proof as in Proposition 31 using the negation of ~ in lieu of 7^, and Pspace 



hardness follows. □ 

We proceed with the last two fragments of the second class. 

Proposition 33. In full generality, the class of problems {3, A, V}-FO(^) exhibits dichotomy: if the core 
of S> has one element then the problem is in L, otherwise it is NP-complete. As a corollary, the class of 
problems {3, A, V, =}-F0(^) exhibits the same dichotomy. 



In our preliminary work |Mar08 Mar061, the proof of the above is combinatorial and appeals to 



Hell and Nesetfil's dichotomy theorem for undirected graphs [HN90|. An alternative proof of this result 



also appeared in | |BHR09| (and to a lesser extent in |H R09] ). We will give here an algebraic proof 



which uses a variant of the Galois connection Inv — End due to Krasner |Kra38| for {3, A, V, =}-F0. 
The variant for {3, A, V}-FO involves hyper-endomorphisms rather than endomorphisms because of the 
absence of equality. A hyper-endomorphism of ^ is a function from B to the power-set of B that is total 



and preserving (see Definition 2.5 1. Our purpose is to provide both a self-contained proof and a gentle 
introduction to the techniques we shall use for the fragment {3, V, A, V}-FO. 

For a set F of hyper-endomorphisms on the finite domain B, let Inv(F) be the set of relations on B 
of which each / in F is an hyper-endomorphism (when these relations are viewed as a structure over B). 
We say that S G lnv(7^) is invariant or preserved by (the hyper-endomorphisms in) 7^. Let hE(=^) be the 
set of hyper-endomorphisms of ^. Let (■^){3,a.v}-fo be the set of relations that may be defined on ^ in 
{3,A,V}-F0. 

Lemma 34. Let r := {r^,. . . ,rk)be a k-tuple of elements of SS. There exists a formula Q^''^"^^ , . . . , U]^ G 
{3, A, V}-FO such that the following are equivalent. 
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(i) (=^,r;,...,r[)|=0j^'^-^^-"°(i.i,. ..,«,). 

(ii) There is a hyper-endomorphism from rj , . . . , r^-) to , . . . , 

Proof, let s := (Z?! , . . . ,b\g\ ) an enumeration of the elements of ^ and (p^[r,s) (vi , • • • , V|b| ) be the associ- 
ated conjunction of positive facts. Set 

("1, •••,"*:) :=3vi,...,V|B| <p,<^(r_s)(Vl,...,V|e|). 

The forward direction is clear as the witness s[,..., s'^^^ for vi , . . . , V|b| provides a hyper-endomorphism 
/ defined as /(^,) 9 5- and /(r,) 9 rj. 

For the backwards direction, one may build an endomorphism from ri,...,rk) to r'j , . . . , r[) 
from the given hyper-endomorphism. The result follows from the implication from (i) to (iv) of Proposi- 
tion H □ 

Theorem 35. For a finite structure .% we have (■^){3,a.v}-fo = lnv(hE(=^)). 

Proof. Let (p(v) be a formula of {3, A, V}-FO with free variables v. We denote also by <p(v) the relation 
induced over SS. 

1. <p(v) G ('^){3,A,v}-FO =^ 'P(v) G lnv(hE(,^)). This is proved by induction on the complexity of 
(p(v). 

(Base Cases.) When (p(v) := /?(v), the variables v may appear multiply in R and in any order. 
Thus R is an instance of an extensional relation under substitution and permutation of positions. 
The result follows directly from the definition of hyper-endomorphisms. 

(Inductive Step.) There are three subcases. We progress through them in a workmanlike fashion. 
Take/G hE(=:^). 

(a) (p(v) := v^(v) A i/a'(v). Let v := (vi, . . . ,v/). Suppose =^ |= <p(;ci, ... then both =^ |= 

. . . and |= i//'(xi, . . . ,x/). By Inductive Hypothesis (IH), for any y\ G fix\\. . . , 
yi e f{xi), both ^ 1= Yiyi ,yi) and ^ 1= Y'iyi , • • • ,yi), whence ^ |= (^(yi , . . . ,3;/). 

(b) (p{\) := v^(v) V v^'(v). Let V := (vi, . . . ,v/). Suppose =^ ^ ... then one of ^ |= 

■ ■ ,xi) or ^ 1= i//^'(xi, . . . ,xi); w.l.o.g. the former. By IH, for any yi G f{x\),. ..,yi^ 
f{xi),.^^ Y{yu---,yi), whence^ ^ (p{yi, . . . ,yi). 

(c) (p(\) := (//(vjw). Let v := (vi,...,v/). Suppose ^ \= 3w yi^i, ■ ■ ■ ,xi,w); then for 
some x', ^ v^(xi, . . . ,x/,x'). By IH, for any yx G /(xi),...,j/ G /(x/),/ G /(x'), ^ |= 
W[y\.---.yuy'), whereupon.^ ^ 3w (//"(ji, . . . ,3;/,^). 



2. 5G lnv(hE(<^)) ^ 5g (^){3^ ,A,v}-FO- Consider the A;- ary relation 5 G lnv(hE(=^)). Letri 
be the tuples of S. Set Q^'^"'^^ , . . . , U]^ to be the following formula of {3, A, V}-FO: 



5 • • • ) • m 



q{3,A,V}-F0/ n,, V , /i{3,A,V}-FO/ 



For r,- := (r;i , . . . , r,i), note that r,i , . . . , r,-*:) |= 0rf ''^•^^'^"(mi , . . . , w^t). That 65(^1 , ...,Uk)=S 
now follows from Part (ii) of Lemma 34 since S G lnv(hE(,^)). 

□ 

Note that in | HR091 it is erroneously claimed that endomorphisms (not hyper-endomorphisms) are the 
correct algebraic object for the fragment {3, A, V}-FO - this is not correct and only holds for the richer 
fragement {3, A, V, =}-F0. 
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Corollary 36. Let SS and l^' be finite structures over the same domain B. If hE(^) C hE{^') then 
{3, A, V}-FO(=^') <L {3, A, V}-FO(=^). 



Proof. If hE(^) C hE(=:^')'then lnv(hE(^')) ^ lnv(hE(=^)). From Theorem 35 it follows that (^'){3,a.v}-fo 
C (=^){3.A,v}-FO- Recalling that contains only a finite number of extensional relations, we may there- 
fore effect a Logspace reduction from {3, A, V}-FO(=^') to {3, A, V}-FO(^) by straightforward substi- 
tution of predicates. □ 

of Proposition \33\ By Proposition [5} we may assume w.l.o.g. that ^ is a core. This means that every 
hyper-endomorphism of ^ is in fact an automorphism - we identify hyper-endomorphisms whose range 
are singletons with automorphisms - and thus hE(i^) is a subset of S„ where n = \D\. If D has one ele- 



ment, then the problem is trival. If D has two elements, then hE(i^) C hE(=^NAE) = •^2- By Lemma 36 
it follows that {3, A, V}-FO(=^nae) <l {3, A, V}-F0(^). Since the former is a generalisation of the 
NP-complete CSP(^nae)> the latter is NP-complete. If ^ has n>2 elements, we proceed similarly 
with Ji^n. □ 

Proposition 37. In full generality, the class of problems {3, A, V, 7^}-F0(^) exhibits dichotomy: if 
\D\ = 1 then the problem is in L, otherwise it is NP-complete. Consequently, the fragment extended with 
= follows the same dichotomy. 



Proof. The proof is similar to that of Proposition 31 Let \D\ = n. The inequality symbol ^ allows to 



simulate Jt^„. When n = 2, using disjunction we may simulate =^nae- NP-completeness follows by 
reduction from CSP(^) when n>3 and from CSP(,^nae) when n = 2. Note that equality is not used 
in our hardness proof and may trivially be allowed when \D\ = 1. Thus, the classification is the same 
whether one allows = or not. □ 

All fragments of the second class follow a natural dichotomy. 

Corollary 38. For any syntactic fragment J£ of FO in the second class, the model checking problem 
££{Sl^ is trivial {in Lj when the J£-core of Si has one element and hard otherwise (NP-complete for 
existential fragments, Pspace-complete for fragments containing both quantifiers). 

3.3 Third Class 

Proposition 39. In full generality, the problem {3, A, 7^}-F0(^) is in L if \D\ = I, in P if \D\ = 2 and 
3 is bijunctive or afftne, and NP-complete otherwise. The fragment extended with = follows the same 
dichotomy. 

Proof. We classify first the fragment extended with =. When \D\ > 3, we may use ^ to simulate 
CSP(J^£)|) which is NP-complete. When \D\ = 1 the problem is trivially in L. We are left with the 
Boolean case. Let S^ denote the extension of S with 7^. Note that {3, A, 7^}-F0(^) coincides with 
{3, A}-F0(^^) which is the Boolean CSP(^^). We apply Schaefer's theorem. The relation / is nei- 
ther Horn, nor dual-Horn, nor 0- valid nor 1 -valid as it is not closed under any of the following Boolean 
operations: A, V, cq or ci (the constant functions and 1). The relation 7^ is both bijunctive and affine 
as it is closed under both the Boolean majority and minority operation (see Chen's survey for the defi- 



nitions |Che09|). Consequently, {3, A,/}-FO(^) is in P if ^ is bijunctive or affine and NP-complete 
otherwise. 

Note that we have not used = in the hardness proof when \D\ > 3. When \D\ = 2, we appeal to 



Schaefer's theorem (Theorem 28 1, the proof of which relies on the Galois connection Pol — Inv which 
assumes presence of =. However, the hardness proofs in Schaefer's theorem rely on logical reductions 
from {3, A}-FO(=^nae)> which use definability of ^nae {3,A}-F0. Hence, our claim follows for 
the fragment {3, A, /}-F0. □ 
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Proposition 40. In full generality, the problem {3, V, A, /}-F0(^) is in L ;/ \D\ = 1, in P if \D\ = 2 
and S> is bijunctive or afftne, and Pspace-complete otherwise. The fragment extended with = follows the 
same dichotomy. 



Proof. This is similar to Proposition 39 When \D\ > 3, we may use ^ to simulate QCSP(^£)| ) which is 



Pspace-complete. In the Boolean case, we apply Theorem 29 to {3, V, A}-FO(^^) and the result follows. 



Again equality is not used to prove hardness and the result follows for the fragment without =. □ 

The case of {3,A}-F0 and {3,A,=}-F0 almost coincide as equality may be propagated out by 
substitution, and every sentence of the latter is logically equivalent to a sentence of the former, with the 
exception of sentences using only = as an extensional predicate like 3xx = x which are tautologies as we 
only ever consider structures with at least one element. In the case of {3, V, A, =}-F0, some equalities 
like 3;c3)';c = y and \/x3yx = y may also be propagated out by substitution. However, equalities like 
3x\/yx = y and \/x\/yx = y can not, but they hold only in structures with a single element. This technical 
issue does not really affect the complexity classification, and it would suffice to consider {3, A}-FO and 
{3,V,A}-F0. The complexity classification for these four fragments remain open and correspond to 
the dichotomy conjecture for CSP and the classification program of the QCSP. In practice, we like to 
pretend that equality is present as it provides a better behaved algebraic framework, without affecting 
complexity. 

This leaves the fragment {3, V, A, V}-FO from our fourth class, which we deal with in the remainder 
of this paper. 



4 Tetrachotomyof {3,V,A,V}-FO(^) 

The following - left as a conjecture at the end of |MM12a MM 10] - is the main contribution of this 
paper. Recall first that a shop / over a set D is an A-shop if there is an element m in D such that f{u) = D; 
and, that / is an E-shop if there is an element xoiD such that ^ (x) = D. 

Theorem 41. Let Sj be any structure. 

\. If & is preserved by both an A-shop and an E-shop, then {3, V, A, V}-FO(^) is in L. 

II. If S> is preserved by an A-shop but is not preserved by any E-shop, then {3, V, A, V}-FO(^) is 
NP-complete. 

III. If Si is preserved by an E-shop but is not preserved by any A-shop, then {3, V, A, V}-FO(^) is 
co-NP-complete. 

IV. If Q) is preserved neither by an A-shop nor by an E-shop, then {3,V, A,V}-FO(^) is Pspace- 
complete. 



Proof. The upper bounds (membership in L, NP and co-NP) for Cases I, II and III were known from |MM12a |, 
but we reprove them here as a corollary of Theorem 19 to keep this paper self-contained. 

Note that an A-shop is simply a ?7-X-shop with IJ = {u}, for some u in D, and X C D. We may 
therefore replace every universal quantifier by the constant u and relativise every existential quantifier to 
X by Theorem 19 This means that {3,V, A, V}-FO(^) is in NP when it has an A-shop as a surjective 
hyper-endomorphism. 

Note that an E-shop is simply a U-X-shop with X = {x} for some x in D, and U C D. So Case III is 
dual to Case II and we finally turn to Case I. 
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With both an A-shop and an E-shop, we have a U-X-shop with U = {u} and X = {x} where u and 
X are in D. We may therefore replace every universal quantifier by the constant u and every existen- 
tial quantifier by the constant x, by Theorem [l9| We have reduced {3, V, A, V}-FO(^) to the Boolean 
sentence value problem, known to be in L | |Lyn77[ . 

Theorem |46] deals with the lower bounds. NP-hardness for Case II and co-NP-hardness for Case III 
are proved in Subsection|4.3.3[ Pspace-hardness for Case III is proved in Subsection |4. 3 .41 □ 



4.1 Methodology : the Galois Connection Inv — shE 



The results of this subsection (§ 4. 1 1 appeared in | MM 12a | and are proved here to keep the present paper 
self-contained. 

Let shE(^) be the set of surjective hyper-endomorphisms of Let (^){3,v,a,v}-fo be the sets of 
relations that may be defined on ^ in {3, V, A, V}-FO. 

Lemma 42. Letr: = [ri,. . . ,ri() be a k-tuple of elements of ,% . There exists a formula gj^'^''^'^'' . . . ^ui^) E 

{3, V, A, V}-FO such that the following are equivalent. 

f\ I I /^ I /i{3,V,A,V}-FO/ n 

(l) (=!^,ri,...,r^) 1= ' ' (Ml,..., Mi). 

(ii) There is a surjective hyper-endomorphism from ri, . . . ,rk) to r'j , . . . , r^). 

Proof. Let r € B'^, s:= {bi,. . . , b\B\ ) be an enumeration of B and t G b'^' . Recall that (p,^(r,s) ("i i • • • ) "/t> vi , . . . , V|b| ) 
is a conjunction of the positive facts of (r,s), where the variables (u,v) correspond to the elements (r,s). 
Similarly, <p,?^(r,s,t)("i t ■ ■ :Uk,v\, . . . ,v\g\,w\, . . . ,w\b\) is the conjunction of the positive facts of (r,s,t), 
where the variables (u,v,w) correspond to the elements (r,s,t). Set oj^'^'^'^^ ■ ■■ ,Uk) '■= 



3vi, . . . ,V[fi| <P^(r,s)(Ml, • . . ,Mi:,Vi, . . . ,V|b|) A Vwi . . .W\b\ 

V *P^(r,s,t) (mi , . . . , Myt, Vl , . . . , V[B[ , Wl , . . . , >V[B[ ) . 

[Backwards.] Suppose / is a surjective hyper-endomorphism from ri , . . . , r^;) to (^', , . . . , 
where SS' := (we will wish to differentiate the two occurrences of SB). We aim to prove that |= 
qJ3,v,a,v} pOj^/^^ _ ^ Choose arbitrary s\ G f{bi),. . . G f{b\s\) as witnesses for vi, . . . ,V|5|. Let 
t' := (fj, . . . ,f|g|) G B'l^l be any valuation of wi, . . . ,w\b\ and take arbitrary ?i, . . . ,t\g\ s.t. t[ G f{t\), . .., 
^\B\ ^ fi^\B\) (here we use surjectivity). Let t := (fi, . . . It follows from the definition of a surjective 
hyper-endomorphism that 

^ 1= <P,«(rs)('"l,--->^/t,'^l,---,'^|B|) A<P,«(r,s,t)(^l,- 

[Forwards.] Assume that ^' |= . . . ,r[), where SS' := SS. Let b[, . . . ,b'^j^^ be an 

enumeration of B' := B^ Choose some witness elements s[,. ..,s'^^ for vi, . . . ,V|g[ and a witness tuple 
t:=(fi,...,f|B|)GBl^l s.t. 

(t) ^' h (P3g(r,s] (r'l , . . . , r[, ^1 , . . . , ) A q),gg(r,s,t) {r[,---,r'k,s\,...,s\g\,b[,..., b\s\). 

Consider the following partial hyper-functions from B to B' . 

"'One may imagine b\,.. ., b\^^\^ and fe'j , . . . , to be tiie same enumeration, but this is not essential. In any case, we will 
wish to keep the dashes on the latter set to remind us they are in iM' and not 
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1. /r given by fr{ri) := {r^}, for \ <i<k. 

2. f, given by U{bi) = {s^}, for 1 < / < (totality) 

3. ft given by b\ G ft{bj) iff ti = bj, for 1 <i,j< \B\. (surjectivity) 

Let / := /r U/s U/t; / is a hyper-operation whose surjectivity is guaranteed by ft (note that totality is 
guaranteed by /s). That / is a surjective hyper-endomorphism follows from the right-hand conjunct of 

(t). □ 

Theorem 43. For a finite structure we have (^){3,v,a,v}-fo = lnv(shE(=^)). 

Proof 1. <?>(v) G (^){3 v,A,v}-F0 =^ <p(v) G lnv(shE(=^)). This is proved by induction on the com- 
plexity of <p(v). We only have to deal with the case of universal quantification in the inductive 
step, the other cases having been dealt with in the proof of the Inv — hE Galois Connection. 

(Inductive Step continued from proof of Theorem [35]) 

(d) <p(v) := Vw (//(Vjw). Let V := (vi, . . . ,v/). Suppose |= Vw ^{x\,. . . ,x/,w); then for each x', 
1= \^{x\^. . . ,x/,x'). By IH, for any y\ G /(xi), . . . ,y/ G /(x/), we have for all / (remember 
/ is surjective), SS |= (/^(ji, . . . jj/,/), whereupon |= Vw (/^(ji, . . . ,3'/,w). 

2. 5 G lnv(shE(^)) =^ 5 G (^){3,v,a,v}-fo- Consider the fc-ary relation S G lnv(shE(=^)). Let 
ri , . . . , Tm be the tuples of S. Let Q^^"^"^^ ^°(mi , . . . , m^:) be the following formula of {3, V, A, V}-FO: 

^{3,V,A,V}-F0. X ^{3,V,A,V}-FO. s 

For r,- := (r,i , . . . , r,*.), note that r,i , . . . , |= ,...,Uk) (viewing the identity 

endomorphism as a surjective hyper endomorphism). That 0^^'^'^'^^ . ..,Uk) = S now fol- 



lows from Part (ii) of Lemma 42 since 5 G lnv(shE(=^)). □ 

□ 

Corollary 44. Lef ^ and J^' be finite structures over the same domain B. If s\\E{3S) C s\\^{3S') then 
{3,V, A, V}-FO(^') <L {3,V, A, V}-FO(=^). 



Proof If shE(=^) C shE(^'). then lnv(shE(^')) ^ lnv(shE(=:^)). From Theorem |43| it follows that 
y,A,v}-FO ^ ('^){3,v,A,v}-FO- Recalling that contains only a finite number of extensional rela- 
tions, we may therefore effect a Logspace reduction from {3, V, A, V}-FO(^') to {3, V, A, V}-FO(=^) by 
straightforward substitution of predicates. □ 

Consequently, the complexity of {3, V, A, V}-FO(^) is characterised by shE(^). 
4.2 The Boolean case 



We recall the case \B\ = 2 (from |MM12a|), with the normalised domain B := {0, 1} as a warm-up. It 



may easily be verified that there are five DSMs in this case, depicted as a lattice in Figure 4.2 The two 
this lattice that represent the two subgroups of ^2 are drawn in the middle and bottom. We 
for the shop that sends to {0, 1} and 1 to { 1}. 



write 






01 


1 


1 



Theorem 45 ( | |MMI2a[ ). Let be a boolean structure. 









1 


01 



I. If either -r— y- or -r— is a surjective hyper-endomorphism of SS, then {3,V, A, V}-FO(,^) is in 
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Figure 1 : The boolean lattice of DSMs with their associated complexity. 



11. Otherwise, {3,\/,A,\/}-F0{l^) isPspace-complete. 



4.2 



If shE(=^) contains 



01 

1 r 



then 



Proof. shE(=^) must be one of the five DSMs depicted in Figure 

we may relativise every existential quantifier to 1 and every universal quantifier to by Theorem 1 8 and 
evaluate in L the quantifier-free part. The case of 









1 


01 



is similar with the role of and 1 swapped. 
We prove that if shE(^) = ( ) then {3, V, A, V}-FO(^) is Pspace-complete. The structure 
has DSM shE(^) = ( ° | ' ). It suffices therefore to prove that {3, V, A, V}-F0(J^2) is Pspace-hard, 



If follows from Corollary 



since (A^)C( 
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1 
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1 



which we did by reduction from QCSP(=^nae) in the proof of Proposition 31 

that when shE(=^) = ( 



, {3,V,A,V}-F0(=^) is also Pspace-hard 

□ 



4.3 Proving Hardness 

Our aim is to derive the following lower bounds. 

Theorem 46. II. If & is preserved by an A-shop but is not preserved by any E-shop, then {3,y,/\,\/}-F0{!^) 
is NP-hard. 

III. If & is preserved by an E-shop but is not preserved by any A-shop, then {3, V, A, V}-FO(^) is 
co-NP-hard. 

IV. If & is preserved neither by an A-shop nor by an E-shop, then {3, V, A, V}-FO(i^) is Pspace-hard. 



It follows from Proposition 26 and Corollary 20 that the complexity of a structure ^ is the same as 
the complexity of its U-X-core. Hence in this Section, we assume w.l.o.g. that U UX = D. We will say 
in this case that the DSM is reduced. This is the critical ingredient, hitherto missing, that is needed 
to obtain the full classification. In order to prove Theorem [46] we need to establish the following: 



II. If U is of size one and X of size at least two then {3, V, A, V}-FO(^) is NP-hard; 
III. If X is of size one and U of size at least two then {3, V, A, V}-FO(^) is co-NP-hard; and. 
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IV. If both U and X have at least two elements then {3, V, A, V}-FO(i^) is Pspace-hard. 

In the following, we will describe a DSM ^ as being (NP-, co-NP-, Pspace-)hard in the case that 
{3, V, A, V}-FO(^) is hard for some & G Inv(^). In order to facilitate the hardness proof, we would 
like to show hardness of a monoid ^ with a very simple structure of which ^ is in fact a sub-DSM 
is the completion of As in general .r#^reserves fewer relations than ^ , the hardness of ^ would 
follow. We would like the structure of ^ to be sufficiently simple for us to build canonically some 
gadgets for our hardness proof. Thus, we wish to better understand the form that elements of ^ may 
take. In order to do so, we first define the canonical shop of ^ to be the U-X shop h in ^ , guaranteed 



by Proposition 26 with the property that \h{z)\ is maximal for each z € U\X. Note that this maximal 



h is unique, as given hi and hi of the form in Proposition 26 h\ ohi is also of the required form, and 



further satisfies \hi o/j2(z)| > |/zi(z)|, |/J2(z)|» for all z G ?7 \X. 
4.3.1 Characterising reduced DSMs 

Any U-X-shop in ^ will be shown to be in the following special form, reminiscent of the form of the 
canonical shop. 

Definition 47. We say that a shop f is in the 3-permuted form if there are a permutation ^ ofX DU, a 
permutation % ofX \ U and a permutation vofU\X such that f satisfies: 

• for any y in U HX, f{y) = { C ()')}; 

• for any xinX\ U, f{x) = {x{^)}'< <^nd, 

• for any u in U\X, f{u) = {u(m)} UX„, where X„ C X \ U. 

Lemma 48. If a shop f satisfies f{X) n (?7 \X) = then f is in the 3-permuted form. 



Proof. The hypothesis forces an element of X to reach an element of X and Lemma 23 forces two 



elements of X to have different images. Since X is finite, there exists a permutation j8 of X such that for 



every x in X, f{x) = {^{x)}. Since Lemma 21 forces in particular an element of U to have at most one 

element of U in its image and since U is finite, it follows that there exists a permutation aofU such that 

forevery Mint/, /(M)n[/ = {«([/)} and f-\u)r\U = {a-\U)]. 

It follows that there exists a permutation ^ of [/ nX such that for any y inU r\X, f{y) = {^(y)}- 
The existence of a permutation xofX\U such that j3 is the disjoint union of % and follows. Hence, 

for any xmX\U, f{x) = 

Similarly, there must also be a permutation v ofU\X such that a is the disjoint union of v and l^. 

Hence, for any umU\X, f{u) nU = {v{u)}. Elements of U\X may however have some images in 

X\U. So we get finally that for any uinU\X, there is some C Xj, C X \ ?7 such that f{u) = { u (m) } UX„. 

This proves that / is in the 3-permuted form and we are done. □ 

Theorem 49. Let ^ he a reduced DSM. Every shop in ^ is in the i -permuted form. Moreover, every 
U -X-shop in ^follows the additional requirement that the elements ofU\X cover the set X\U, more 
formally that 

fiu\x)nx= U x, = x\u. 

ueu\x 



Proof. We can now deduce easily from Lemmata 25 and 48 that ?7-X-shops in ^ must take the 3- 



permuted form. It remains to prove that an arbitrary shop / in ^ is in the 3-permuted form. Let h be 
the canonical shop of It follows from Lemma [s] that f:=hofohisa ?7 -X-shop. Hence, /' is 
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in the 3-permuted form. Let z in X and u mU\X. If /(z) B u then f'{z) 9 u and /' would not be in 



the 3-permuted form. It follows that f{X) n (?7 \X) = and appealing to Lemma 48 that / is in the 



3-permuted form. □ 

We do not need the following result in order to prove our main result. But surprisingly in a reduced 
DSM, U and X are unique. This means that we may speak of the canonical shop of ^ instead of some 
canonical ?7-X-shop. It also means that we can define the U-X-coxe of a structure ^ without explicitly 
referring toUorX as the minimal substructure of which satisfy the same {3,V, A,V}-FO sentences. 

Theorem 50. Let ^ be a structure that is both a U -X-core and a U'-X'-core then it follows that U = U' 
andX = X'. 

Proof. We do a proof by contradiction. Let h and h' be the canonical U -X-shop and ?7'-X'-shop, respec- 
tively. Assume U' and let x mU' \ U . Note that since D = U VJX, our notation is consistent as x 
does belong toX\U . Thus, there exists some u'mU \X such that h{u) 3 {u,x} (and necessarily u / x). 



By Theorem 49 h has to be in the 3-permuted form w.r.t. U' and X' , which means that h can send an 
element to at most one element of U' . Since x belongs to U' , it follows that u belongs to D\U' =X' \ U' . 
But the three permuted form prohibits an element of X' to reach an element of U' . A contradiction. 

It does not follow yet that X' = X as the pairs of sets may have shifting intersections. However, the 
dual argument to the above applies and yields X =X' . □ 

Corollary 51. Let Si be a finite structure. The U -X-core of S is unique up to isomorphism. It is 
a minimal induced substructure S of S, that satisfies the same {3,V, A, V}-FO /ormw/ae with free- 
variables in D. Moreover, once D is fixed, there are two uniquely determined subsets U andX such that 
U L)X = S C D which are minimal within D with respect to the following equivalent properties, 

• S has\/U-3X-re\ativisationw.rt. {3, V, A, V}-FO; o?; 

• S has a U -X-shop that may act as the identity over UDX. 



Proof. The last point follows from our definition of a U -X-core and from Proposition 26 It is equiv- 



alent to the yU-3X-relativisation property by Theorem 19 It follows that S and S satisfy the same 



{3, V, A, V}-FO formulae with free-variables in D (see Corollary 20 1. Conversely, if S and S satisfy the 



same {3, V, A, V}-FO formulae with free-variables in D, then S has D-D-relativisation. The existence 



of a "D-D-shop" follows by Theorem 19 Enforcing the minimality criteria, we get some ?7 -X-shop 



with some U.X C D (this is because, we may proceed by retraction, as explained in the beginning of 
Subsection 2. 10 1. Moreover, by minimality of S, we must have U UX = D. We have a ?7 -X-core as in 



our original definition in terms ofaU -X-shop satisfying minimality criteria. It follows from Theorem 50 



that U and X are unique (within D). □ 

Recall that the {3, V, A, V}-FO-core S' of S is the smallest (w.r.t. domain size) structure that is 
{3, V, A, V}-FO-equivalent to S. 

Proposition 52. The notion of a U -X-core and of a {3, V, A, V}-FO-core coincide. 

Proof. Let S be a structure that is a ?7 -X-core with (unique) subsets U and X. Let c be the canonical 
shop of S. 

Let S' be a {3, V, A, V}-FO-core of S, that is a smallest (w.r.t. domain size) structure that is 
{3,V, A, V}-FO-equivalent to S. Let U' and X' be subsets of D' witnessing that S' is a U'-X' core. 
Note that U' UX' = D' by minimality of S' (and consequently, U' and X' are uniquely determined by 



Theorem 50 1. Let c' be the canonical shop of S'. 
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By Proposition 10 since S) and S)' are {3, V, A, V}-FO-equivalent, there exist two surjective hyper- 
morphisms g from ^ to 3)' and / from Qi' to 

Let U" be a minimal subset of (g)^' such that g{U") = U'. Note that /o c' og is a ?7"-surjective 
shop of By minimality of ?7, it follows that \U\ < \U"\ < \U'\. A similar argument over gives 
l^^'^'l < l^^'^l. and consequently, \U\ = \U'\. Moreover, since c o [f o c' o g) is a ?7"-surjective X-total 



surjective hyperendomorphism of ^, By Theorem 50 it follows that U = U" . 



This means that there is a bijection a' from U' to U such that, for any u' in U' , g ^ {u') = {a'{u')}. 
By duality we obtain similarly that and that there is a bijection j3 from X to X' such that, for 

any xmX, g{x) = {^{x)}. 

Thus, g acts necessarily as a bijection from U r\Xto U' HX'. 

The map g from D to D' defined for any uinU as g{u) := a'^^ (u) and g{x) := j3 (x) is a homomor- 
phism from ^ to that is both injective and surjective. 

A symmetric argument yields a map / that is a bijective homomorphism from to Isomorphism 
of ^' and ^ follows. 

□ 

Remark 2. To simplify the presentation, we defined the ^-core as a minimal structure w.r.t. domain 
size. Considering minimal structures w.r.t. inclusion, we would get the same notion for {3, V, A, V}-FO. 
This is also the case for CSP, but it is not the case in general. For example, this is not the case for the 



logic {3, V, A}-FO, which corresponds to QCSP ^MM12bl 



Lemma 53. Let ^ he a reduced DSM with associated sets U andX. There are only three cases possible. 

1. Ur\X^%U\X^%andU\X^%. 

2. U=X. 

3. unx = (i). 

Proof. We prove that J7 C X is not possible. Otherwise, letxmX\U and h be the canonical shop. There 
exists some uinU C X such that h{u) 9 x by ?7-surjectivity of h. Since u does not occur in the image of 
any other element than u under the canonical shop, this would mean that hisX\ {«} -total, contradicting 
the minimality of X. 

By duality X C J7 is not possible either and the result follows. □ 



4.3.2 The hard DSM above ^ 

Define the completion ^ of to be the DSM that contains all shops in the 3-permuted form of 
More precisely, the canonical shop of ^ is the shop h where every set X„ is the whole set X\U, and, for 
every permutation (^ofXn?7,;^ofX\J7 and V of U\X, any shop in the 3-permuted form with these 
permutations is in Note that by construction, ^ is a sub-DSM of Note also that the minimality 
of U and X still holds in We will establish hardness for whereupon hardness of ^ follows from 
Theorem l43l 



4.3.3 Cases II and III: NP-hardness and co-NP-hardness 



We begin with Case II. We note first that U = {u} and |X| > 2 implies ?7nX = by Lemma 53 The 
structure W the disjoint union of a clique of size |X| with an isolated vertex u, has associated 

DSM ^ The problem {3, A, V}-FO(J^|x| W ^) is NP-hard, since the core of J^^l is J^\x\ by 



Proposition 33 
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For Case III, we may assume similarly to above that X = {x}, \U\>2 and U CiX = (Z)hy Lemma 53 
We use the duality principle, which corresponds to taking the inverse of shops. Since the inverse of 
an {x}-total ?7-surjective shop with ?7 > 2 is a {?7}-total {;c}-surjective shop, we may use the structure 
Jfpi^jei which is {V, V, A}-FO-equivalent to (and {V, V, A}-FO(^^^\) is co-NP-hard). 

4.3.4 case IV: Pspace-hardness 



We assume that |?7| > 2 and |X| > 2 and consider the tree possible cases given by Lemma 53 



Case 1: when [/\X/0andX\[//0 Recall that if ^ is a sub-DSM of a hard DSM ^ 

then ^ is also hard (see Theorem |43|). 

We write UAX as an abbreviation for {X\U)U {U \X). To build ^ from we added all permu- 
tations, and chose for each set X„ = X \ U. We carry on with this completion process and consider the 
super-DSM which is generated by a single shop g' defined as follows: 

• for every y inXnU, g'{y) := XAU ; and, 

• for every z in XAU, g'{z) :=XnU, where XAU denotes {X\U)U {U\X). 

The complete bipartite graph J€xAu.xnu has for DSM. Observing that there is a full surjective 



homomorphism from J(^xAU,xnu to thus by Proposition 10 the two structures agree on all sentences 
of {3,V, A,V,-i}-FO and so also on all sentences of {3,V, A, V}-FO. It suffices therefore to prove that 
{3,V, A,V}-FO(^) is Pspace-hard, which we did by reduction from QCSP(,^nae) in the proof of 
Theorem l45l 



Case 2: when U = X The clique has DSM ^ The problem {3,V, A, V}-FO(^(/|) is Pspace- 
complete by Theorem|45]in the Boolean case; and, beyond that, it is also Pspace-hard as a generalisation 
of the Pspace-complete QCSP(^j/|). The Pspace-completeness of {3,V, A,V}-FO(i^) follows from 
Theorem l43l 

Case 3: when U CiX = (d We can no longer complete the monoid ^ into as we would end up 
with a trivial monoid. The remainder of this section is devoted to a generic hardness proof. Assume that 
\U\ = j > 2 and |X| = )t > 2 and w.l.o.g. let [/ = {1,2, . . .7} and X = {j + l,j + 2, . . .j + k}. Recalling 
that the symmetric group is generated by a transposition and a cyclic permutation, let ^ be the DSM 
given by 



1 


2.j+\,...J + k 1 


2,j+l,...,j + k 1 


l,j+l,....j + k 1 


l,j+l,...,j + k 


2 


I, j+l,. ..J + k 2 


3, j+l,..., j + k 2 


2,j+l,...,j + k 2 


2,j+l,...,j + k 


3 


3, j+l,..., j + k 3 


4,j+l,...,j + k 3 


3,j+l,...,j + k 3 


3, j+l,..., j + k 












j 


j, j+l,..., j + k j 


I, j+l,..., j + k j 


j, j+l,..., j + k j 


j, j+l,..., j + k 


j+l 


J+l ' J+l 


J+l ' J+l 


J+2 ' j+l 


J + 2 


J + 2 


J+2 j+2 


J+2 j+2 


J+l J+2 


J + 3 


J + 3 


J + 3 ./ + 3 


J + 3 ./ + 3 


J + 3 ./ + 3 


J + 4 












j + k 


j+k j+k 


j+k j+k 


j+k j+k 


J+l 



We will give a structure ^ such that shE(^) = Firstly, though, given some fixed uinU and x in 

\u\ \x\ 

X, let be the symmetric graph with self-loops with domain D = UUX such that 

• u and X are adjacent; 
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3Q 



. 1 



(a) 'S{ 



2,2 




(b)<| 



Figure 2: Main Gadget. 




(c) 'SI 



\v\\x\ 



• The graph induced by X is a reflexive clique 'J^x^', and, 

* U\ {«} and X \ {x} are related via a complete bipartite graph ^x\{jc}|.|c/\{h}|- 
The structure ^i^' '^' and the more specific 'iS'^''^ are drawn in Figure jij Denote by £"1^'''^' the binary 



relation of '^u} and let ^ be the structure with a single 4-ary relation with domain D = U UX 
specified as follows. 



U U ("'^) 

ueU \ ^xex 



U.X 



u 



U {XUX2) 



£|c/|.W 



Essentially, when the first argument in a quadruple is from U, then the rest of the structure allows for 

\u\ \x\ 

the unique recovery of some ^u.x ; but if the first argument is from X then all possibilities from X for 
the remaining arguments are allowed. In particular, we note from the last big cup that {xi,X2,xs,X4.) is a 
tuple of R^ for all quadruples xi,X2,xs,X4 in X. 

Lemma 54. shE(#) = ^ 



Proof. Recall that, according to Theorem 49 and our assumption on U, X and a maximal (w.r.t. 
sub-shop inclusion) shop /' is of the following form, 

• for any x in X \ ?7 = X, /(x) = {;t;(x)}; and, 

• forany Mint/ \X = [/,/(«) = {u(m)}UX. 



where x and v are permutations of X and U, respectively. 

(Backwards; C shE(^).) It suffices to check that a maximal shop /' in 
holds by construction. We consider first tuples from (xi,X2) x E, 



preserves ^. This 



u\,\x\ 



A tuple with elements from X only will map to a like tuple, which must occur, so we can ignore 
such tuples from now on. 

A tuple (xi,X2,M,X3) maps either to {x{xi),X{^2),viu),x{x3)) which appears in {x{m),X{x2)) x 

\u\ \x\ 

^w(h) x{^3)' maps to a tuple containing only elements from X. 

A tuple (xi , X2 , X3 , m) maps either to (;|^ (xi ) , (xa) , (x3 ) , u (m)) , which appears in (;|^ (xi ) , (X2)) x 

\u\ \x\ 

^v{u) x{x3)' '^^ maps to a tuple containing only elements from X. 
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\u\ \x\ 

We consider now tuples from {u,x) x EuJ 



• If the first coordinate u is mapped to v{u), then the tuple is mapped to different tuples from 

,\x\ 



{y{u),x{xj) X El|^^^^yl^\■, depending whether the second u is mapped to an element from X or to 



v{u). 

• Otherwise, the first coordinate u is mapped to an element x\ from X, and some other element from 
u' in U occurs (or the tuple contains elements from X only) and a tuple is mapped to a tuple of the 
form {x]_,x{x),v{u'),xi,) which appears in {xi,x{x)) ^E^yi^S)^^- 

(Forwards; shE(^) C ^ .) We proceed by contraposition, demonstrating that 7?^ is violated by any 
/ ^ We consider the different ways that / might not be in ^ . 

• If / is s.t. u G fix) for ;c G X and u^U then we, e.g., take (m,;c,x,x) G but (z, u, u, u) ^ (for 
any z. G /(«)) and we are done. It follows that f{X) = X. 

• Assume now that /is s.t. {x'^Xj} C f{x) forx\ andx,Xj,X2 GX. Let u,u' ^Uhe s.t. u' G /(«). 
Take {u,x,u,x) G R^; [u' ,x\,u' jx'j) ^ R^ and we are done. It follows that / is a permutation x on 
X. 

• Assume now that / is s.t. {u\,u'2} ^ /(«) for u[ / u'2 and u,u[,u'2 G U. Let x,x^ G X be s.t. 

x' G f{x). Take {u,x,u,x) G (mj,x',M2)-^') ^ and we are done. It follows that / restricted 
to ?7 is a permutation v on U. 

Hence, / is a sub-shop of a maximal shop /' from the DSM and / belongs to ^ (recall that a DSM 
is closed under sub-shops). The result follows. □ 

Proposition 55. {3,V, A,V}-FO(^) is Pspace-complete. 

Proof. We begin with the observation that {3, V, A, V}-FO(§^„„v ) is Pspace-complete (for each u^U 
and X G X). This follows straightforwardly from the Pspace-completeness of {3, V, A, V}-F0(^|^3^), 



\u\ \x\ 

the simplest gadget which is depicted on Figure 2a These gadgets ^u,x agree on all equality-free 



sentences - even ones involving negation - by Proposition 10 as there is a full surjective homomorphism 
from ^fj^''''^' to'^if. 

We will prove that {3,\/ , A,\/}-FO{^^f) is Pspace-hard, by reduction from the Pspace-complete 
problem QCSP(^nae)- Recall that we may assume w.l.o.g. that universal variables are relativised to U 
and that existential variables are relativised to X, by Theorem[T9[ Let (p be an instance of QCSP(^nae)- 
We reduce (p to a (relativised) instance y of {3, V, A, V}-FO($^j 3 ). The reduction goes as follows: 

• an existential variable 3x of cp is replaced by an existential variable 3v_x G X in i/a; 

• a universal variable Vm of cp is replaced by Vm G ?7 3v„ G X, ^(mjVh) in i//^; and, 

• every clause Q := R{a,p, 7) in cp is replaced by the following formula in i/A, 

Vc; G U, E{ci,Va)V E{ci,vp)V E{ci,Vy). 

The truth assignment is read from 3 choices in X for the variables v: we arbitrarily see one value in X as 
true and the other as false. It is not relevant which one is which for the problem not-aU-equal satisfiability, 
we only need to ensure that no three variables involved in a clause can get the same value. The Vc, G U 
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acts as a conjunction, enforcing "one of Va,Vj3,Vy is true" and "one of VajVfjjVy is false". This means 
that at least one in three has a different value. 

Now, wecan prove that {3, V, A, V}-FO(^) is Pspace-complete by substituting /?(mo,-^0)")V) for each 

instance of E{u,v) in the previous proof, and by quantifying the sentence so-produced with the prefix 

\u\ \x\ 

Vmo £ U 3x0 £ X, once uo and xq are chosen, play proceeds as above but in the copy "^uo-xo , and the 
result follows. □ 



4.4 The Complexity of the Meta-Problem 



The {3, V, A, V}-FO((T) meta-problem takes as input a finite a-structure & and answers L, NP-complete, 
co-NP-complete or Pspace-complete, according to the complexity of {3,V, A, V}-FO(^). The principle 
result of this section is that this problem is NP-hard even for some fixed and finite signature Go, which 
consists of two binary and three unary predicates (the unaries are for convenience, but it is not clear 
whether a single binary suffices). 

Note that one may determine if a given shop / is a surjective hyper-endomorphism of a structure ^ 
in, say, quadratic time in \D\ Since we are not interested here in distinguishing levels within P, we will 
henceforth consider such a test to be a basic operation. We begin with the most str aightforward case. 

Proposition 56. On input the question "is {3, V, A, V}-FO-(^) in L?" is in P. 



Proof. By Theorem 41 we need to check whether there is both an A-shop and an E-shop in shE(^). 
In this special case, it suffices to test for each m,x in D, if the following {M}-{x}-shop / preserves 
f{u) :=D and f-^{x):=D. □ 

Proposition 57. For some fixed and finite signature Oq, on input of a <j -structure 'S, the question "is 
{3, V, A, V}-FO(^) in NP (respectively, NP-complete, in co-NP, co-NP-complete)?" is NP-complete. 

Proof. The four variants are each in NP. For the first, one guesses and verifies that has an A-shop, for 



the second, one further checks that there is no {M}-{.x}-shop (see the proof of Proposition 56 1. Similarly 
for the third, one guesses and verifies that ^ has an E-shop; and, for the fourth, one further checks that 



there is no {M}-{x}-shop. The result then follows from Theorem 41 

For NP-hardness we will address the first problem only. The same proof will work for the second 
(for the third and fourth, recall that a structure ^ has an A-shop iff its complement ^ has an E-shop). 
We reduce from graph 3-colourability. Let ^ be an undirected graph with vertices V := {vi,V2, . . . ,Vi}. 
We will build a structure over the domain D which consists of the disjoint union of "three colours" 
{0, 1,2}, u, and the "vertices" from V. 

The key observation is that there is a structure whose class of surjective hyper-endomorphisms 
shE(^^y) is generated by the following A-shop: 



fv-.-- 









1 


1 


2 


2 


H 


0, 1,2, it, V'l .... . 


V'l 


0,1,2 


V2 


0,1,2 








0,1,2 



The existence of such a '^y is in fact guaranteed by the Galois connection, fully given in fMarlO], but 
that may require relations of unbounded arity, and we wish to establish our result for a fixed signature. 
So we will appeal to Lemma 58 below, for a ay-structure with the desired class of surjective hyper- 
endomorphisms, where the signature Oy consists of one binary relation and three monadic predicates. 
The signature Gq will be ay together with a binary relational symbol E. 
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The structure S^cg is defined as in 'Sy for symbols in ay, and for the additional binary symbol E, as 
the edge relation of the instance of 3-colourability together with a clique ^ for the colours {0, 1,2}. 
By construction, the following holds. 

• Any surjective hyper-endomorphism g of will be a sub-shop of fy. 

• Restricting such a shop g to V provides a set of mutually consistent 3-colourings: i.e. we may 
pick arbitrarily a colour from g(v,) to get a 3-colouring g. If there is an edge between v,- and vy 
in , then Eivi^v j) holds in ^c^. Since g is a shop, for any pair of colours c,-,cy, where c; € ^(v,) 
and Cj G ^(v;), we must have that E{ci,Cj) holds in =5%. The relation E is defined as over the 
colours. Hence / cy and we are done. 

• Conversely, a 3-colouring g induces a sub-shop g of fy: set g as fy over elements from {0, 1,2, m} 
and as g over V. The detailed argument is similar to the above. 

This proves that graph 3-colourability reduces to the meta-question "is {3, V, A, V}-FO(^) in NP". □ 

Note that it follows from the given proof that the meta-problem itself is NP-hard. To see this, we take 
the structure o^c^ from the proof of Proposition|57]and ask which of the four classes L, NP-complete, co- 
NP-complete or Pspace-complete the corresponding problem belongs to. If the answer is NP-complete 
then §^ was 3-colourable; otherwise the answer is Pspace-complete and $^ was not 3-colourable. 

Lemma 58. Let Oy be a signature involving one binary relations E' and three monadic predicates 
Zero, One and Two. There is a Oy-structure such that shE(^^y) = (/y). 



Proof. We begin with the graph ^' on signature (£'), depicted on Figure 3a Note that 



shE(^) := 



We now replace c by {0, 1,2} and v by V to obtain a graph . Formally, this graph is the unique graph 




c 

' — ^(y 

(a) 0" (b) ^" 

Figure 3: Building a structure with shE(^^y) = (/y). 

with domain {0, 1 , 2, m} U V such that the mapping which maps {0, 1 , 2} to c, fixes u and maps V to v, is 
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a strong surjective homomorphism. By construction, 



shE(^") := ( 






0,1,2 


1 


0,1,2 


2 


0,1,2 


u 


0, \,2,u,vi,. . 




Vl 


0,1,2 






V.S 


0,1,2 



We now build as the structure with binary relation E' which is the edge relation from and by 
setting the unary predicates as follows: Zero holds only over 0, One holds only over 1 and Two holds 
only over 2. This effectively fixes surjective hyper-endomoiphisms to act as the identity over the colours 
{0,1,2} as required. □ 



5 Conclusion 

We have classified the complexity of the model checking problem for all fragments of FO but those 



corresponding to the CSP and the QCSP. Our results are summarised as Figure 4.4 The inclusion of 
fragments is denoted by dashed edges, a larger fragment being above. Each fragment is classified in 
two fashions. Firstly, we have indicated on the figure the notion of core used to classify fragments, by 
regrouping them in the same box. Secondly, we have organised the fragments in four classes according 
to the nature of the complexity classification they follow. The first class is trivial. Tractability for a 
fragment ^ of the second class corresponds precisely to having a one element ^-core. The third class 
regroups fragments which have a non trivial classification viz complexity, in the sense that it does not 
always depend on the size of the ^-core, and include the two open cases of CSP and QCSP which we 
discuss in some detail below. The fourth class contains the fragment {3,V, A,V}-FO which exhibits a 
behaviour intermediate between the third class and the fourth class: its complexity is fully explained in 
terms of the U-X-core., yet as this notion involves two sets, the fragment exhibits richness in its ensuing 
tetrachotomy. 

For the CSP, the dichotomy conjecture has been proved in the Boolean case by Schaefer (see Theo- 
rem!: 



28 1 and in the case of undirected graphs. 



Theorem 59 ( | HN90| ). Let he an undirected graph. IfS is hipartite then CSP{^) is in L, otherwise 



CSPC^) is NP-completeQ 



For CSP in general, it would suffice to settle the dichotomy conjecture for (certain) directed graphs | FV98[ 



The dichotomy conjecture has been settled for smooth digraphs (graphs with no sources and no sinks) |BKN09 1 
According to the algebraic reformulation of the dichotomy conjecture, it would suffice to prove that ev- 



ery structure that has a Sigger's term has a tractable CSP (see |BV08 BuUl | for recent surveys on the 
algebraic approach to the dichotomy conjecture). 

For the QCSP, much less is known. We have already seen that a dichotomy between P and Pspace- 
complete holds in the Boolean case (Theorem [29]). However, the complexity is not even known for 



undirected graphs. It is fully classified for graphs with at most one cycle. 
Theorem 60 ( | |MM06| ). Let he an undirected graph. 



• If^ is bipartite then QCSP{^) is in L; 

'^In the bipartite case, assuming that the graph has at least one edge, then the core of 5f is Jf^2. The problem CSP(^) is 
2-colourability which is in the complexity class symmetric logspace now known to be equal to L IReiOSI. 
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• if^ is not bipartite and not connected then QCSP{^) is NP-complete; and, 

• j/^ not bipartite, connected and contains at most one cycle then QCSP{^) is Pspace-complete. 

The algebraic approach to QCSP uses surjective polymorphisms and has led to a trichotomy in the 
case where all graphs of permutations are available. Recall first the definition of some special surjective 
operations. A k-aiy near-unanimity operation f satisfies 



f{xi,...,Xk) 



\x ii{xi,...,Xk] = {x];2LnA, 
1 X if all but one of xi , . . . , X|(- is equal to x. 

When ^ = 3, we speak of a majority operation. The k-ary near projection operation is defined as 



lk{xu...,Xk) 
The ternary switching operation is defined as 



x\ when |{xi, ... ,Xyt}| = ^; and, 
X/t otherwise. 



s{x,y,z) -- 

The dual discriminator operation is defined as 



X if y = z, 
y if X = z, 
z otherwise. 



y if y = z; and, 
X otherwise. 



d[x,y,z) 

When f{x,y,z) = x — y + z w.r.t. some Abelian group structure, we say that / is an afftne operation. 



Theorem 61 ( | BBC''"09 1). Let & be a structure such that there is an extensional binary symbol for each 
graph of a permutation ofD. Then the complexity of QCSP { !^) follows the following trichotomy. 

• If & has a surjective polymorphism which is the dual discriminator, the switching operation or an 
affine operation then QCSP{&) is in P. 

• Else, ifl\D\ ci surjective polymorphism of then QCSP{&) is NP-complete. 

• Otherwise, QCSP{&) is Pspace-complete. 

In general, it is known that if a structure ^ is preserved by a near-unanimity operation then QCSP(^) 
is in P, because it implies a property of collapsibility. This property means that an instance holds if, and 
only, if all sentences induced by keeping only a bounded number of universal quantifiers - the so-called 
collapsings - hold | Che08[ . 

For undirected partially reflexive graphs (i.e. with possible self-loops), we have the following partial 
classification (reformulated algebraically). 

Theorem 62 ( |Marll| ). Let 3^ be a partially reflexive forest. 

• If ,9' is ^/\}-VO-equivalent to a structure that is preserved by a majority operation then 
QCSP{^) is in P; and, 

• otherwise, QCSP{^) isNP-hard. 

In the case of structures with all constants, Hubie Chen has ventured some conjecture regarding the 
NP/Pspace-hard border: he suggests that the polynomially generated power property (PGP) - a property 
which generalises collapsibility - explains a drop in complexity to NP(see |Chel2| for details). 
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